Website owners increasingly ask whether AI assistants can discover and cite their content, and whether they should publish an llms.txt file. The important distinction is between crawler access, search indexing, and a product’s decision to use or cite a source. These are related but separate systems. There is no universal file or setting that guarantees a website will appear in AI-generated answers.
What robots.txt does
The robots.txt file is a public set of crawl instructions placed at the root of a host, for example https://example.com/robots.txt. Compliant crawlers may use its user-agent groups and allow/disallow rules to decide which paths to request. Robots.txt is not access control: private or sensitive content must be protected with authentication and proper server permissions. A blocked URL may still be discovered from links, and blocking crawling does not reliably remove a URL from search results.
Robots.txt, indexing and AI visibility are different
Allowing a crawler to fetch a page does not guarantee that a search engine indexes it or that an AI service will cite it. Indexing depends on many factors, including page availability, content quality, canonical signals and the search product’s policies. AI services may use different crawlers for search retrieval, user-initiated browsing, model training or other purposes. Their names, purposes and controls can change, so check each provider’s current official documentation before changing rules.
Should you add llms.txt?
llms.txt is a proposed convention for providing language-model systems with a curated guide to a website’s important resources. It is not the same standard as robots.txt, and it should not be treated as a Google ranking factor or a guaranteed route into AI answers. Adoption and interpretation vary by tool. If you publish one, consider it optional documentation that may help some systems understand your site—not a replacement for accessible pages, navigation, sitemaps or clear content.
Build a safe crawler policy
Start by listing the crawlers relevant to your goals and checking the provider’s current documentation. Decide whether you want to permit search indexing, user-triggered retrieval, training-related crawling or other access; these may be controlled by different agents and policies. Avoid copying a generic list of user-agent names from an old blog post because names and product behaviour can change.
Before editing robots.txt, download or copy the current file and note what each rule is intended to do. Keep rules as narrow as practical. A broad Disallow: / rule can block an entire host for matching compliant crawlers. Rules for CSS, JavaScript or image resources can also interfere with rendering if they block resources needed to understand the page. Test the resulting file and inspect important URLs after deployment.
Example: a cautious robots.txt starting point
This illustrative example allows general crawling and points to a sitemap. It is not a universal recommendation: your site’s existing rules, private areas, staging setup and provider-specific preferences may require different directives.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap_index.xmlReplace the example domain and sitemap path with the real values for your website. Do not add disallow rules for WordPress administration or other paths by habit; check what those paths expose and how your site uses them. The robots.txt file cannot protect account pages or unpublished information from people who know the URL.
How to validate the setup
- Open the live robots.txt URL and confirm it returns the intended content.
- Check for duplicate or contradictory user-agent groups and broad disallow rules.
- Confirm the sitemap URL is correct and the sitemap lists canonical pages intended for indexing.
- Use Google Search Console’s current inspection and indexing reports to investigate Google crawling and indexing; do not assume one test represents every AI crawler.
- Review server logs, when available, to understand which agents request which pages and whether they receive errors.
- Recheck the provider’s official crawler documentation periodically, especially after policy or product changes.
Improve the content that systems can understand
Technical access is only one part of visibility. Publish original, accurate, useful information; use descriptive page titles and headings; make authorship and business identity clear; cite primary sources for important claims; keep key information in crawlable page text; and link related pages naturally. Avoid making unsupported promises that a particular formatting trick, schema type or llms.txt file will secure citations. Structured data should accurately describe visible page content and must follow the relevant search engine’s rules.
Common mistakes
- Assuming llms.txt is an official Google ranking or indexing requirement.
- Assuming all AI products use the same crawler or obey the same controls.
- Using robots.txt as a security mechanism for private information.
- Blocking crawling and expecting that alone to remove a URL from search results.
- Allowing every crawler without considering bandwidth, licensing, privacy and business preferences.
- Changing live rules without testing important pages and keeping a rollback copy.
Conclusion
Use robots.txt to express a deliberate crawl policy, not as a shortcut to rankings. Treat llms.txt as an optional emerging convention, and verify any claimed benefit against the current documentation of the services you care about. A technically accessible site, clear content, accurate information and consistent internal architecture remain the dependable foundations for search and potential AI discovery.