llms.txt and AI Crawlers: A Guide for Business Websites

This guide is for business owners, marketing managers and the developers who look after company websites in India and abroad, who keep hearing that AI crawlers are either stealing their content or are the future of search. After reading it you will be able to write an llms.txt file, set robots.txt rules per AI bot, decide whether to block training crawlers, configure CDN bot settings without hurting search visibility, and read your logs to see what AI systems actually do on your site.

What llms.txt is and what goes in it

llms.txt is a proposal published in late 2024 for a Markdown file served at /llms.txt. Where robots.txt tells crawlers where they may go, llms.txt tells language models what the site is about and where the most useful pages live, in a format a model can read in one pass. It is a convention, not a standard: nothing enforces it and nothing penalises its absence.

The file has a fixed shape: an H1 with the company name, a blockquote with a one-paragraph summary, then H2 sections, each a list of links with a short description after each one. A software company might have sections for Services, Products, Locations, Pricing and Documentation. An optional section titled Optional holds links that a model can skip when it is short on context. Keep it under about 200 lines, use absolute URLs, and update it when you add major pages.

llms-full.txt: the long version

llms-full.txt is the companion file that contains the actual content of your key pages in Markdown, concatenated into one document, rather than links to them. Documentation sites use it so a model can load a whole product manual in one request. For a business website, generate it from your CMS at build time if you publish one at all, and keep it to pages you would be happy to see quoted in full.

ai.txt file: the earlier opt-out idea

The ai.txt file was an earlier proposal, from 2023, for declaring whether your text, images and other media may be used for AI training. It never reached wide adoption and most crawlers do not look for it. The same intent is now expressed through robots.txt rules per bot, and more recently through content signal directives that some CDNs add to robots.txt. An existing ai.txt does no harm, but do not rely on it.

AI training vs AI search crawling: two different bots from each company

AI training vs AI search crawling is the distinction most articles miss, and it drives every decision below. A training crawler collects pages to build future models; blocking it has no effect on whether you appear in AI answers today. A search or retrieval crawler fetches pages to answer live questions and cite sources; blocking it removes you from those answers. Most AI companies run both under different user agent names, plus a third type that fetches a page only when a user asks about it.

User agent	Operator	Purpose	Block it?
GPTBot	OpenAI	Training data collection	Your choice
OAI-SearchBot	OpenAI	ChatGPT search index and citations	Allow
ChatGPT-User	OpenAI	Fetches a page when a user asks	Allow
PerplexityBot	Perplexity	Search index and citations	Allow
Perplexity-User	Perplexity	Fetches a page when a user asks	Allow
Google-Extended	Google	Gemini training and grounding control	Your choice
Googlebot	Google	Search, including AI Overviews	Never
ClaudeBot	Anthropic	Training data collection	Your choice
CCBot	Common Crawl	Open web archive used for training	Your choice
Bytespider	ByteDance	Training data collection	Usually block

GPTBot robots.txt rules

GPTBot robots.txt rules work the same way as any other user agent block: a User-agent line naming GPTBot, followed by Allow and Disallow lines. OpenAI states that GPTBot respects robots.txt and publishes its IP ranges. The useful pattern for a business site is to disallow GPTBot site-wide, or from pricing and proprietary content, while leaving OAI-SearchBot and ChatGPT-User allowed. If you want ChatGPT to cite your pages but not train on them, that split is what the two names are for.

PerplexityBot and Perplexity-User

PerplexityBot is the indexing crawler for Perplexity's answer engine; Perplexity-User is the fetcher that retrieves a page when someone asks a question that needs it. The company publishes IP ranges for verification. Because Perplexity shows sources prominently and sends measurable referral traffic to the pages it cites, blocking PerplexityBot is a direct loss for most business sites. The exception is content you sell access to, where a cited summary would replace the purchase.

Google-Extended: what it does and does not control

Google-Extended is a robots.txt token, not a separate crawler. Googlebot does the crawling; Google-Extended tells Google whether the pages it fetched may be used to train Gemini models and to ground Gemini answers. Disallowing Google-Extended does not affect Google Search rankings, and according to Google it does not affect inclusion in AI Overviews or AI Mode, which draw on the normal search index. If you block Googlebot itself to keep content away from AI, you leave search entirely, which is almost never the trade you want.

Should you block AI crawlers? A decision for business websites

Should you block AI crawlers is really three questions: which bots, which pages, and what you expect to gain. For a company selling services or products, visibility inside AI answers is a growing source of enquiries, and training crawlers are a separate matter of principle rather than revenue. Our default advice to clients: allow every search and user-triggered fetcher, decide on training bots according to your view of the trade-off, and block known scrapers that ignore rules anyway through the CDN rather than robots.txt.

AI crawler access by page type

• Service pages, product pages, location pages, about and contact: allow all search and user fetchers. These are the pages you want quoted with your name attached.
• Blog posts and guides: allow search fetchers. Decide on training bots site-wide rather than per post.
• Pricing pages: allow search fetchers if your prices are public anyway; disallow training bots if you change prices often and do not want old figures repeated.
• Paid content, courses, downloadable reports: disallow everything except Googlebot, and put the full text behind login so robots rules are not the only barrier.
• Admin, API, search result pages, cart and account: disallow all bots, AI and otherwise.

Content licensing for AI

Content licensing for AI is becoming a market of its own. Large publishers sign agreements that pay for training access, and some CDN providers now offer pay-per-crawl, where crawlers are charged per request at a price the site sets. For a small or mid-sized business the practical version is simpler: a terms page stating what automated use is permitted, the same preference in robots.txt per bot, and records of crawler activity in case a licensing route opens later. This is not legal advice; speak to a lawyer if your content is the product.

AI crawlers and server load: crawl budget, CDN and protection

AI crawlers and server load became a real operational problem in 2025. Several training crawlers fetch far more aggressively than search engines, hitting every URL including filter combinations and archives, and some ignore Crawl-delay. On a small VPS, a burst of a few thousand requests in a minute can push response times past the point where Googlebot sees timeouts. The sections below keep the bots you want fast and the bots you do not want out.

Crawl budget and AI bots

Crawl budget for AI bots differs from Google's crawl budget. Google sets a request rate per site based on server health and demand; AI crawlers each set their own and do not coordinate. The pages that waste the most are faceted URLs, calendar archives, internal search results and tag pages. Disallow those paths for all user agents, return 404 rather than soft 200 for missing pages, and make sure your sitemap lists only canonical URLs so well-behaved crawlers spend their requests on pages that matter.

AI crawlers and CDN

AI crawlers and CDN caching interact well when the site is cached at the edge: a cached page costs your origin nothing, so even an aggressive crawler is harmless. Problems start when pages bypass the cache: logged-in views, session cookies, or dynamic pages with no cache headers. Set long cache lifetimes on public pages, strip tracking query parameters at the edge so they do not create cache misses, and let the CDN handle rate limiting instead of your application server.

Cloudflare AI bot settings

Cloudflare AI bot settings are the most common place where a business site accidentally disappears from AI answers. The dashboard offers one toggle that blocks known AI crawlers, and since mid-2025 new sites block AI training bots by default. That toggle treats search fetchers and training bots alike unless you refine it. If you use it, add allow rules for the search and user-triggered agents you want, verify them by IP range rather than user agent string alone, and check that the managed challenge page is not being served to crawlers with a 200 status, because a challenge page indexed as content is worse than no page at all.

AI scraping protection beyond robots.txt

AI scraping protection beyond robots.txt is needed because robots rules are voluntary. Crawlers that ignore them can be slowed with rate limits per IP range, blocked by known bad user agents, challenged with a JavaScript test, and starved of value by keeping premium content behind login. Do not block everything that lacks a browser signature: search crawlers, link preview fetchers and uptime monitors all look like bots. Verify the major crawlers by reverse DNS or published IP ranges and allow those explicitly.

Reading AI crawler logs

AI crawler logs are the only way to know what is happening rather than what the documentation says. Pull the last 30 days of access logs, group by user agent, and count requests, bytes and distinct URLs for each. Then check which AI agents are visiting at all, whether they fetch the pages you want cited or waste requests on filter URLs, and whether any get error or challenge responses. A weekly summary by agent tells you within a month whether an AI search engine has started paying attention to your site.

What we have seen on jktechhub.com and client sites in Rajkot

JK Tech Hub publishes its own llms.txt alongside the sitemap. The honest result so far: we see AI search fetchers in our logs and referrals from ChatGPT and Perplexity to service pages, and we cannot prove the file caused any of it. It costs ten minutes a quarter to keep current, which is why we still do it.

The more valuable lesson came from our own hosting. An anti-bot layer on the live site was serving a content-free JavaScript challenge page with a 200 status whenever request bursts crossed a threshold. Crawlers that hit the threshold received an empty page and indexed it. We found it only by reading logs and comparing them with cached copies in search results. Rate limiting is now tuned so real crawlers never trigger it, and we check it monthly.

For a bearings manufacturer in Rajkot, the problem was the opposite. Their hosting provider had enabled a blanket AI bot block by default, which also caught the search fetchers. Enquiries arriving from AI assistant conversations stopped for six weeks before anyone connected the two. Allowing the search agents explicitly restored them within days.

How to decide

If you sell services or products and want enquiries, allow the AI search and user-triggered fetchers, publish an llms.txt, and spend your effort on clear, quotable pages rather than on blocking. If your content is the product, block training and search crawlers from the paid parts, keep them behind login, and allow search engines on the public marketing pages only. If your server is struggling, fix caching and rate limits at the CDN before touching robots.txt, and never let a protection rule serve a challenge page as a successful response. Review your crawler logs quarterly; the bot landscape changes faster than any article can track.

Related reading and next step

• Generative engine optimization services: getting cited in ChatGPT, Perplexity, Gemini and AI Overviews.
• Generative engine optimization guide: how AI answers choose sources.
• SEO audit checklist: robots, sitemaps, canonicals and crawl health in one list.
• AI SEO services in Rajkot: AI visibility work for businesses in Saurashtra and Gujarat.
• SEO services in Rajkot: monthly technical and local SEO from our Mavdi office.

Send your domain through the contact page or WhatsApp +91 7265004040 and we will reply with a crawler log review, a draft llms.txt and a fixed quote within two working days.