robots.txt Generator
Build a correct robots.txt file without touching documentation, crawler rules, optional AI bot blocking, and your sitemap directive, ready to copy.
Build a correct robots.txt file without touching documentation, crawler rules, optional AI bot blocking, and your sitemap directive, ready to copy.
Save this as robots.txt and upload it to the root of your site so it's reachable at yoursite.com/robots.txt.
Choose your default crawler rule, add any paths you want blocked, decide whether AI training bots may access your content, and paste your sitemap URL. Click Generate and copy the finished file. Upload it to your site's root directory, it must be reachable at yoursite.com/robots.txt exactly, not in a subfolder.
The robots.txt file gives crawling instructions to well-behaved bots. It tells search engines which parts of your site to crawl and which to skip, and it points them to your sitemap. Two things it does not do: it doesn't remove pages from Google's index (blocked pages can still appear in results if other sites link to them, use a noindex tag for that), and it doesn't provide security (malicious bots ignore it entirely, so never rely on robots.txt to hide sensitive content).
There's a genuine trade-off. Allowing crawlers like GPTBot means your content can inform AI models, which increasingly means being mentioned or cited when people ask AI assistants questions in your topic area, a growing discovery channel. Blocking them protects your content from being used as training data but removes you from that channel. Search crawlers like Googlebot are unaffected either way; the toggles above only touch training-specific bots. For most sites trying to grow, leaving them open is the pragmatic call. For sites whose content is the product, blocking is reasonable.