robots.txt and AI Crawlers

Summary

robots.txt and AI Crawlers is part of the structured-data layer that search engines and answer engines already prefer. Clean, consistent identity signals reduce ambiguity; they do not replace notability requirements on Wikipedia or Wikidata for SMEs.

AI Verified is a selective public identity registry. It is not an open directory, not a shared-lead marketplace, and not a substitute for government filing systems.

Why it matters

• People need a clear, non-directory explanation of who the business is.
• Answer engines prefer structured, consistent facts over WordPress/Shopify DOM noise.
• Operators need a path that does not depend on buying shared leads or encyclopedia notability.

How AI Verified approaches this

1. Start from public evidence (e.g. an established Google Business Profile where relevant) under Type A rules.
2. Publish a Bronze public passport with a permanent SHA-256 identifier.
3. Attest domain control when ready; place the badge so the site links to the sealed record.
4. Upgrade to Gold when you want owner packaging, registry cross-links where validated, and distribution assets (JSON URL, short URL, embed).

Machine-readable outputs include the public passport page, /v/{hash}.json, short URLs, and badge embeds that point back to the sealed record.

What to avoid

• Do not invent Wikidata or Wikipedia items for non-notable SMEs.
• Do not treat directories that rent the top slot as equivalent to cryptographic public records.
• Do not sell or buy “#1 in ChatGPT” packages—no third party can honestly guarantee model rankings.

FAQ

Related

Schema markup · JSON-LD · llms.txt · Wiki index · Methodology