Free Robots.txt Generator Tool, Build a robots.txt Online

Free Tool

Robots.txt Generator

Configure your crawler rules using the form below. The live output updates in real time, download your robots.txt or copy it directly into your site's root directory.

Robots.txt controls crawling, not indexing. A blocked page can still appear in search if other sites link to it, and blocking it stops Google reading the very noindex tag that would remove it. To keep a page out of search, use noindex, not Disallow.

Configure Crawler Rules

Quick Presets

User-agent Rules

Bot / User-agent Directive

Disallowed Paths (applies to rules above)

Disallow Paths one per line, e.g. /admin/

Allow Paths override specific disallowed paths

Options

Crawl-delay

Seconds between bot requests, use with care, not supported by Googlebot

Block AI Training Bots

Blocks GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, Bytespider and more

Block SEO Crawlers

Blocks AhrefsBot, SemrushBot, MJ12bot, DotBot, reduces server load from scrapers

Block Social Media Bots

Blocks Twitterbot, facebookexternalhit, LinkedInBot, Pinterest crawlers

Sitemap URLs

robots.txt Output

🤖

Configure rules on the left and click
Generate robots.txt

How It Works

Perfect Robots.txt in 4 Steps

Choose a Preset

Start from a proven template, Standard SEO, WordPress, E-commerce or Block All. Or build from scratch with custom rules.

Configure Bot Rules

Add as many User-agent rules as needed. Choose which bots to allow or block and specify which paths they can or cannot access.

Add Sitemaps & Options

Reference your XML sitemap URLs and enable options like crawl delay. The live preview updates as you type.

Download & Upload

Click Download to save robots.txt, then upload it to the root of your domain, e.g. https://yoursite.com/robots.txt

What is a robots.txt file?

A robots.txt file is a text file at your site root that tells crawlers which parts of your site they may request. It lives at yoursite.com/robots.txt, and a robots.txt generator writes the rules for you so the syntax is right first time.

The rule everyone gets wrong: it controls crawling, not indexing. Disallow tells a crawler not to fetch a page. It does not tell Google to keep it out of search. A disallowed page can still rank if other sites link to it, and because Google cannot fetch it, it never sees the noindex tag that would actually remove it. Blocking is not hiding.

It is also only a request. Well-behaved crawlers obey it. Scrapers and email harvesters read it as a map of what you would rather they did not see.

Written by Chirag Arora, digital marketing consultant with 17+ years of hands-on SEO experience across 500+ projects. He designed this tool and the other 22 free tools on this site. Verify on LinkedIn.

Reference

Which AI crawlers should you allow?

Crawler	Who runs it	What blocking it costs you
GPTBot	OpenAI, training	Your content stops feeding future ChatGPT models. Blocking does not remove what it already has
OAI-SearchBot	OpenAI, search	You disappear from ChatGPT search results. This is the one people block by accident
ClaudeBot	Anthropic	Same trade as GPTBot, for Claude
PerplexityBot	Perplexity	You lose citations from the engine that links out most generously
Google-Extended	Google, AI training	You opt out of Gemini training. Normal Google Search ranking is unaffected
CCBot	Common Crawl	An open dataset many models train on. Blocking it is broad and blunt

There is a real decision here and it is not the same for everyone. A publisher protecting paid work has good reason to block training bots. A business that wants customers to find it through AI has the opposite interest, because blocking the crawler removes you from the answers. The trap is that training and search are different bots: block GPTBot and you leave training, block OAI-SearchBot and you vanish from ChatGPT search. Most people who block one meant the other. Google's robots.txt documentation covers the syntax, and you can check what a rule does in Search Console before you publish it.

Expert Tips

Use Robots.txt Like a Pro SEO

⚠️

Robots.txt ≠ Security

Robots.txt is a polite request, not a security measure. Any bad actor can ignore it. Never rely on robots.txt to hide sensitive data. Use authentication and server-level access control for that.

🔗

Always Reference Your Sitemap

Add a Sitemap: directive pointing to your XML sitemap. Even though you can submit it in Google Search Console, having it in robots.txt ensures every bot that visits discovers it automatically.

🔍

Test Before Going Live

Use Google Search Console's robots.txt Tester to verify your rules before uploading. A single misplaced Disallow: / will block Google from indexing your entire site, always test first.

Two wildcards give the file real power, and both are easy to get wrong. The asterisk matches any run of characters, so Disallow: /*.pdf blocks paths that contain .pdf. The dollar sign anchors a rule to the end of a URL, so Disallow: /*.pdf$ blocks files that end in .pdf but leaves report.pdf?ref=x alone. Paths are case-sensitive, so /Admin/ and /admin/ are different rules. And robots.txt is only one of three controls that people mix up. Robots.txt manages crawling. The meta robots tag in a page's head manages indexing for that one page. The X-Robots-Tag HTTP header does the same job as meta robots but for files with no head to edit, like PDFs. If you need a page kept out of the index, use noindex, not a Disallow, because a blocked page can never be crawled to see the noindex in the first place.

About robots.txt

What Does robots.txt Do?

The robots.txt file sits at the root of your domain and uses the Robots Exclusion Protocol to tell compliant web crawlers which pages they are allowed or not allowed to access. Search engine bots like Googlebot, Bingbot and others check this file before crawling your site. Common uses include blocking admin pages, preventing duplicate content indexing, saving crawl budget on large sites and referencing XML sitemaps. Remember, it controls crawling, not indexing, a page can still appear in search results if other pages link to it, even if it's disallowed.

Free Robots.txt Generator

A Free Robots.txt File Generator Built From Real Campaigns

This free robots txt file generator builds the file for you online and previews the exact text before you upload it. Pick your platform, choose what to block, decide how to treat AI crawlers and copy the result. It runs in the browser with no account and nothing to install, and it works for any website whether that is WordPress, Blogger, Shopify or hand-built HTML.

Robots.txt is the smallest file on your site and the one that does the most damage when it is wrong. Across 500+ projects I have seen a single stray Disallow: / survive a staging launch and quietly remove an entire site from Google. Nobody noticed for six weeks. It is four characters. That is why this tool previews the file rather than just handing you a download, you should read the thing before it goes live.

The newer decision is what to do about AI crawlers, and most generators still pretend they do not exist. This one includes GPTBot, ClaudeBot, Google-Extended and the rest, because whether you want your content feeding AI answers is now a real business question rather than a technical footnote. Once the file is live, generate your sitemap with the free sitemap generator and add it as the last line, then check whether AI actually mentions you with the free AI visibility checker. The rest are in the free SEO tools hub.

One distinction saves people from a classic mistake. This file controls crawling, not indexing, so blocking a page here does not remove it from Google, a disallowed URL can still sit in results with no snippet. To keep a page out of the index you use a noindex meta tag or an X-Robots-Tag header, and the page has to stay crawlable for Google to see that instruction. Google stopped honouring a Noindex line inside robots.txt back in 2019, so it does nothing here. The file itself follows the RFC 9309 standard, must return a 200 status, and stays under 500 kilobytes for Google. And never list private paths as a way to hide them, that just hands attackers a map, protect those with real access controls instead.

Built by Chirag Arora
Digital marketing expert, 17+ years of hands-on SEO experience across 500+ projects. Connect on LinkedIn

Chirag Arora presenting technical SEO and crawl control at an industry event

More Free Tools

All Tools, Zero Cost

FREE TOOL

Keyword Research Tool

Discover thousands of keyword ideas with volume, difficulty, CPC and intent data, powered by Google Autocomplete.

Launch Tool

FREE TOOL

Meta Tag Generator

Generate perfectly optimised title tags, meta descriptions, Open Graph and Twitter Cards in seconds.

Launch Tool

FREE TOOL

Website Speed Checker

Measure Core Web Vitals, LCP, INP, CLS, with prioritised recommendations to fix every issue.

Launch Tool

FAQ

Robots.txt Generator FAQ

Common questions about this free tool, answered clearly.

Does robots.txt stop a page from being indexed?

No. This trips a lot of people up. Robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other pages link to it. To keep a page out of the index, use a noindex tag instead.

Where does the robots.txt file go?

It must sit at the root of your domain, at yoursite.com slash robots.txt. Search engines only look there. A robots.txt in a subfolder is ignored.

Should I block AI crawlers in robots.txt?

Only if you do not want your content used or cited by AI tools. Blocking crawlers like GPTBot or Google-Extended removes you from those answers. If you want AI visibility, allow them and block only private areas.

Do I need a robots.txt generator for WordPress or Blogger?

WordPress serves a virtual robots.txt automatically, and Yoast or Rank Math let you edit it from the dashboard, so you may not need a file at all. Blogger has a custom robots.txt setting in Search Settings. Use this generator to write the rules correctly, then paste them into whichever box your platform gives you. The syntax is the same everywhere. What differs is where you put it.

Should I block AI crawlers like GPTBot in robots.txt?

It depends on what you sell, and the honest answer is that most people get this backwards. If you publish content people pay for, blocking training bots is reasonable. If you want customers to find you through AI, blocking the crawler removes you from the answers, which is the opposite of what you wanted. The critical detail is that training and search are separate bots. GPTBot is training, OAI-SearchBot is ChatGPT search. Block GPTBot and you keep your ChatGPT visibility. Block OAI-SearchBot and you disappear from it.

Is this better than the Small SEO Tools robots.txt generator?

For writing the file, they do the same job. The difference is that this one includes current AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and Google-Extended, and it previews the exact file before you copy it. Most robots.txt generators were written before AI crawlers existed and have not been updated. Beyond that there is not much to choose between them, because it is a text file.

Why Work With Me

Why Hire Me for Your Technical SEO

This free tool writes the file. If you would rather have someone check it is not quietly costing you traffic, here are a few honest reasons businesses hire me as their SEO and digital marketing consultant in India.

The Four Characters That Delete a Site

Disallow: / on a live site removes it from Google. I have found it surviving a staging launch for six weeks while everyone blamed the algorithm. I check the boring file first.

Hands-On, Not Just Oversight

I started as a technical SEO trainee, not in a management seat, and I still write the robots and crawl rules myself. You get direct expertise, not a ticket passed to a developer.

I Built This Tool Myself

The robots.txt generator you just used, I designed it, along with 22 other free SEO tools on this site. It is a direct look at how I set up crawl control before you ever become a client.

AI Crawlers, Decided Deliberately

Blocking GPTBot and blocking OAI-SearchBot do completely different things, and most sites that blocked one meant the other. I make that a decision rather than an accident.

Direct Line to Me

Messages get a reply within 24 hours on business days, and WhatsApp is the fastest way to reach me for anything urgent. You are not waiting in a support queue.

Crawl Budget Where It Counts

On a large site, crawlers waste their time on filters, search pages and parameters. I point them at what earns money instead.

Testimonials

Results From Getting the Basics Right

★★★★★

"Our traffic collapsed and three agencies blamed a core update. Chirag looked at robots.txt in the first ten minutes and found a Disallow left over from the staging site. Recovered in a fortnight."

Founder

SaaS Startup, India

★★★★★

"He explained the AI crawler decision properly instead of just telling us to block everything. We allow search bots and block training bots, which is exactly what we wanted. His 17+ years really shows."

Marketing Head

EdTech Company, India

★★★★★

"As a small business, we started with Chirag's free robots.txt tool, then hired him. Ranked Page 1 for our main terms within 3 months. Practical, no fluff, real value."

Owner

D2C E-commerce Brand

Free Robots.txt Generator & File Builder

Robots.txt Generator

Configure Crawler Rules

robots.txt Output

Perfect Robots.txt in 4 Steps

Choose a Preset

Configure Bot Rules

Add Sitemaps & Options

Download & Upload

What is a robots.txt file?

Which AI crawlers should you allow?

Use Robots.txt Like a Pro SEO

Robots.txt ≠ Security

Always Reference Your Sitemap

Test Before Going Live

What Does robots.txt Do?

A Free Robots.txt File Generator Built From Real Campaigns

All Tools, Zero Cost

Keyword Research Tool

Meta Tag Generator

Website Speed Checker

Robots.txt Generator FAQ

Does robots.txt stop a page from being indexed?

Where does the robots.txt file go?

Should I block AI crawlers in robots.txt?

Do I need a robots.txt generator for WordPress or Blogger?

Should I block AI crawlers like GPTBot in robots.txt?

Is this better than the Small SEO Tools robots.txt generator?

Why Hire Me for Your Technical SEO

The Four Characters That Delete a Site

Hands-On, Not Just Oversight

I Built This Tool Myself

AI Crawlers, Decided Deliberately

Direct Line to Me

Crawl Budget Where It Counts

Results From Getting the Basics Right

Founder

Marketing Head

Owner

Chirag Arora In The News

Speaking, Training & Building Across India

Want a Full Technical SEO Audit Done for You?