Generate your custom robots.txt + llms.txt kit free.
Control crawlers and guide AI models to your content — simple, fast.
.png){kind=small}
.txt files isn’t just about blocking or allowing access.Our kit is designed in three parts — covering search engines, AI training bots, and human triggered bots — so you stay in charge at every level.
We start with the essentials: a solid robots.txt that tells Google, Bing, and other search engines how to crawl your site. This ensures your content is indexed correctly, while also protecting sensitive or irrelevant areas.
Example:
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Disallow: /private/
Next, we address the bots that gather content to train large language models. With clear rules, you can decide if your website should be part of their datasets —or keep it off-limits.
Example:
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Disallow: /private/
Finally, we handle bots triggered by human interactions (like ChatGPT’s browsing, Perplexity, or other AI-powered assistants). These aren’t training, they’re fetching content live. We help you control how they see and use your pages in real time.
Example:
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Disallow: /members-only/
This field lets you define areas of your site that should stay off-limits to all bots. It’s perfect for private zones like /backoffice/, /admin/, or test environments that you don’t want indexed, trained on, or browsed.
Here you can point crawlers and AI systems to your official sitemap. A sitemap helps them understand your site structure and discover the content you actually want highlighted — keeping things organized and improving visibility.
