Start from a proven template, Standard SEO, WordPress, E-commerce or Block All. Or build from scratch with custom rules.
Add as many User-agent rules as needed. Choose which bots to allow or block and specify which paths they can or cannot access.
Reference your XML sitemap URLs and enable options like crawl delay. The live preview updates as you type.
Click Download to save robots.txt, then upload it to the root of your domain, e.g. https://yoursite.com/robots.txt
Configure your crawler rules using the form below. The live output updates in real time, download your robots.txt or copy it directly into your site's root directory.
Configure rules on the left and click
Generate robots.txt
Robots.txt is a polite request, not a security measure. Any bad actor can ignore it. Never rely on robots.txt to hide sensitive data. Use authentication and server-level access control for that.
Add a Sitemap: directive pointing to your XML sitemap. Even though you can submit it in Google Search Console, having it in robots.txt ensures every bot that visits discovers it automatically.
Use Google Search Console's robots.txt Tester to verify your rules before uploading. A single misplaced Disallow: / will block Google from indexing your entire site, always test first.
The robots.txt file sits at the root of your domain and uses the Robots Exclusion Protocol to tell compliant web crawlers which pages they are allowed or not allowed to access. Search engine bots like Googlebot, Bingbot and others check this file before crawling your site. Common uses include blocking admin pages, preventing duplicate content indexing, saving crawl budget on large sites and referencing XML sitemaps. Remember: it controls crawling, not indexing, a page can still appear in search results if other pages link to it, even if it's disallowed.
Discover thousands of keyword ideas with volume, difficulty, CPC and intent data, powered by Google Autocomplete.
Launch ToolGenerate perfectly optimised title tags, meta descriptions, Open Graph and Twitter Cards in seconds.
Launch ToolMeasure Core Web Vitals, LCP, INP, CLS, with prioritised recommendations to fix every issue.
Launch ToolCommon questions about this free tool, answered clearly.
No. This trips a lot of people up. Robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other pages link to it. To keep a page out of the index, use a noindex tag instead.
It must sit at the root of your domain, at yoursite.com slash robots.txt. Search engines only look there. A robots.txt in a subfolder is ignored.
Only if you do not want your content used or cited by AI tools. Blocking crawlers like GPTBot or Google-Extended removes you from those answers. If you want AI visibility, allow them and block only private areas.