Photograph by Adrien Olichon via Pexels. Illustrative image, not a customer or endorsement.
Blog · · 7 min read · NEXUM Team
llms.txt, Schema and AI Crawlers: What Actually Helps AI Visibility in 2026
Short answer: In 2026, the evidence says llms.txt does not help you appear in AI search. Google states that Google Search ignores it, and large log studies show AI crawlers almost never request it. What does help is the unglamorous technical work: letting the right AI crawlers in, serving content they can read in plain HTML, and keeping structured data accurate for its real purpose, which is rich results and clarity. Spend your time there first.
Here is what the data and the official documentation actually say, and a checklist you can use this week.
What is llms.txt?
llms.txt is a proposed Markdown file placed at a site's root (yoursite.com/llms.txt) that summarizes the site and links to key pages for large language models. The idea spread quickly through 2025, and many website platforms now generate one automatically.
The question is simple: does anything important read it?
What Google says
Google's official guide, Optimizing your website for generative AI features on Google Search, addresses it in a "mythbusting" section:
"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."
It adds that creating llms.txt for other services is fine, but "doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
There is one nuance. Search Engine Journal noted that Chrome's Lighthouse added an llms.txt check inside an experimental Agentic Browsing category, aimed at browser agents that navigate sites on a user's behalf. That is about agents understanding site structure, not about search rankings or AI citations.
What the server logs say
Most llms.txt files are never requested
Ahrefs analyzed 137,210 domains and found that 97% of llms.txt files received zero requests in May 2026. Of the small share that were fetched, most requests came from bots, and a notable portion came from SEO tools and researchers checking the files rather than from AI assistants.
AI search crawlers skip it
Elmrank published 14 days of logs across eight sites: crawlers from OpenAI, Google, Anthropic and Perplexity made 126,310 requests and none of them asked for llms.txt, even though most of those sites published one. The same crawlers did request robots.txt thousands of times.
No measurable traffic effect
Search Engine Land tracked 10 sites for 90 days before and after adding llms.txt. Eight saw no measurable change, two saw AI traffic increases that the authors did not attribute to llms.txt, and one declined.
Bottom line: keep llms.txt if your platform generates it and it is accurate, but do not expect it to change your AI visibility.
What about schema and structured data?
Structured data (schema.org markup in JSON-LD) still has value, just not as an "AI hack."
Google's guide says structured data "isn't required for generative AI search, and there's no special schema.org markup you need to add," but recommends continuing to use it as part of your SEO strategy because it helps with eligibility for rich results.
Practical takeaways:
- Use the schema types that match your content: Organization, Product or SoftwareApplication, Article, FAQPage, BreadcrumbList, Review where appropriate.
- Make sure the markup matches what users see. Mismatched or inflated markup is a liability.
- Keep key facts consistent: the name, pricing and plans in your schema should match your pricing page.
Structured data supports clarity. It does not replace content that is worth citing.
What actually helps AI crawlers
1. Let the right bots in
Each AI company documents its own crawlers. Blocking the wrong one can remove you from answers entirely.
- OpenAI: According to OpenAI's crawler documentation, OAI-SearchBot surfaces sites in ChatGPT search, and sites that opt out "will not be shown in ChatGPT search answers." GPTBot is for model training and is controlled separately. ChatGPT-User handles user-triggered fetches.
- Perplexity: PerplexityBot builds Perplexity's search index and Perplexity-User fetches pages for users, as described in Perplexity's crawler docs.
- Google: Google's AI features use its normal Search index, so a page must be indexed and eligible for a snippet to appear in them.
Check three places: your robots.txt, your CDN or firewall bot rules (many block AI bots by default), and your server logs to confirm the bots get 200 responses, not 403s or timeouts.
2. Serve content in readable HTML
If your key content only appears after heavy JavaScript runs, some crawlers may not see it. Google can process JavaScript when it is not blocked, but its guide notes that JavaScript-heavy sites are "generally more complex" for SEO. Server-side rendering or static generation for pricing, product and comparison pages is the safer choice.
Adobe's research gives a sense of the problem. Its Citation Readability Score found that the average U.S. retail homepage scored 63% in September 2026, meaning more than a third of homepage content was not effectively visible to AI systems. Pages that explain things, like guides, FAQs and detailed specs, scored highest.
3. Put important facts in text, not only images
Pricing tables as images, features in carousels, or specs inside PDFs are harder to extract. Put key facts in plain text on the page.
4. Keep a clean, crawlable structure
Logical internal links, an XML sitemap, canonical tags and minimal duplicate content help every crawler, AI or not.
5. Make pages answer questions clearly
Technical access only gets you considered. To be cited, the page must answer the question better than alternatives. Lead with a direct answer, then support it with specifics and original insight.
A 10-point technical checklist for AI visibility
- robots.txt allows OAI-SearchBot, PerplexityBot and Googlebot (decide separately on training bots like GPTBot).
- CDN and firewall rules do not block or challenge AI search crawlers.
- Server logs show AI crawlers receiving 200 responses.
- Key pages are indexed in Google and eligible for snippets.
- Pricing, product and comparison pages render meaningful content without client-side JavaScript.
- Critical facts (plans, prices, features, integrations) are in plain text.
- Schema markup is valid, matches visible content and uses appropriate types.
- XML sitemap is current and submitted in Search Console.
- Facts are consistent across your site, docs and third-party listings.
- llms.txt, if present, is accurate, but not treated as a priority.
How to know if your fixes worked
Technical fixes are only worth it if AI answers change. Track a stable set of buyer prompts across AI assistants before and after your changes, over several weeks, because AI answers vary from day to day. For Google, use the Generative AI performance report in Search Console; for other assistants, you need prompt-based tracking.
NEXUM is a SaaS platform that does this tracking across ChatGPT, Gemini, Perplexity and Claude (Copilot and Grok coming soon). It shows when and how your brand is mentioned, which sources the models cite, and how that changes after you ship improvements, and its GA4 integration connects AI visibility to real visits. You can start free at hellonexum.com with the Explorer plan, or try a paid plan with a 7-day free trial.
FAQ
Does llms.txt improve rankings or AI citations?
There is no evidence that it does. Google says Google Search ignores llms.txt, and studies from Ahrefs and others found that AI crawlers rarely or never request it.
Should I delete my llms.txt file?
Not necessarily. Google says having one will neither harm nor help your Google visibility. If your platform generates it, keep it accurate and move on to higher-impact work.
Do I need special schema for AI search?
No. Google says no special schema.org markup is required for its generative AI features. Standard structured data is still useful for rich results and should match your visible content.
Should I block GPTBot to protect my content?
That is a business decision. OpenAI says GPTBot is for training and is independent of OAI-SearchBot, which controls appearance in ChatGPT search answers. You can block one and allow the other.
How do I check if AI crawlers can access my site?
Review robots.txt, check CDN and firewall bot settings, and search your server logs for user agents like OAI-SearchBot, PerplexityBot and Googlebot to confirm they receive successful responses.
