Does robots.txt block pages from appearing in Google?

Not reliably. A Disallow rule stops crawling, but if other sites link to the page, Google can still index the URL without its content. To remove a page from search results, use a noindex tag or password protection instead.

What is the correct robots.txt for WordPress?

A good WordPress default allows everything except the admin area: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php, followed by your Sitemap line pointing to /sitemap_index.xml or /sitemap.xml. Use the WordPress preset in this generator.

Can I block AI crawlers like GPTBot and ClaudeBot?

Yes. Add a separate group for each AI crawler user-agent (GPTBot, ClaudeBot, CCBot, Google-Extended) with Disallow: /. This generator includes one-click AI crawler blocking.

What happens if I use Disallow: / ?

Disallow: / under User-agent: * blocks every crawler from crawling your entire site. Only use it on staging or development sites — on a live site it can remove your pages from search results.

TechCoff Tools › SEO Tools › Robots.txt Generator

Robots.txt Generator

Create a valid, search-engine-friendly robots.txt file in under a minute. Pick a preset or build your own rules for Googlebot, Bingbot and AI crawlers — then copy or download the finished file. Free, no signup, nothing is uploaded.

✓ 100% Free ✓ WordPress & WooCommerce presets ✓ Block AI crawlers (GPTBot, ClaudeBot) ✓ Built-in URL tester ✓ Runs entirely in your browser

1 · Build your rules

Start from a preset

Sitemap URL

Leave blank if you don't have a sitemap yet. WordPress sites usually use /sitemap_index.xml (Rank Math / Yoast) or /wp-sitemap.xml.

Quick-add common blocked paths

Chips add a Disallow rule to the first (all-bots) group.

2 · Your robots.txt

Live preview

0 lines 0 rules 0 blocked paths 1 bot group(s)

3 · Test a URL against your rules

The tester uses the same longest-match logic as Google: the most specific matching rule wins, and Allow beats Disallow on a tie.

How to use this robots.txt generator

Pick a preset or add rules

Start with WordPress, WooCommerce or SEO-optimized presets, or build your own Allow / Disallow rules for each bot.

Add your sitemap

Paste your sitemap URL so crawlers find every important page faster. One line per sitemap if you have several.

Copy, download & upload

Download the file and place it in your site's root folder so it loads at yourdomain.com/robots.txt. Test a few URLs above before you go live.

What is a robots.txt file?

Robots.txt is a small plain-text file that lives in the root of your website and gives instructions to web crawlers — Googlebot, Bingbot, social media scrapers and AI crawlers — about which parts of your site they may crawl. It follows the Robots Exclusion Protocol and is the very first file a well-behaved bot requests when it visits your domain.

Used well, a robots.txt file keeps crawl budget focused on pages that matter, hides thin or private areas like admin screens, carts and search-result pages, and points bots straight to your sitemap. Used badly — one wrong Disallow: / line — it can remove your entire site from Google. That's why generating and testing the file carefully, instead of typing it by hand, is worth the two minutes.

Robots.txt is not a security tool

Disallowed pages can still be visited by anyone who knows the URL, and bad bots simply ignore the file. Never list secret folders or files in robots.txt — the file itself is public, so it can actually reveal the locations you're trying to hide. Protect private content with passwords or server rules instead.

Robots.txt syntax cheat sheet

Directive	Example	What it does
`User-agent`	`User-agent: Googlebot`	Names the crawler the following rules apply to. `*` means all bots.
`Disallow`	`Disallow: /cart/`	Blocks crawling of a path and everything under it. `Disallow: /` blocks the whole site.
`Allow`	`Allow: /wp-admin/admin-ajax.php`	Re-opens a specific file or sub-path inside a blocked folder. The longest matching rule wins.
`Crawl-delay`	`Crawl-delay: 10`	Asks bots to wait N seconds between requests. Honoured by Bing and Yandex; Google ignores it (use Search Console instead).
`Sitemap`	`Sitemap: https://example.com/sitemap.xml`	Tells every crawler where your XML sitemap lives. Not tied to any user-agent group — goes at the end of the file.
`*` wildcard	`Disallow: /*?`	Matches any sequence of characters — handy for blocking parameter-based duplicate URLs.
`$` anchor	`Disallow: /*.pdf$`	Anchors the match to the end of the URL, e.g. block only URLs that end in `.pdf`.

Ready-made examples — click to load

WordPress (recommended default)

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

WooCommerce store

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=*
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

Block AI training crawlers

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow:

Staging site — block everything

User-agent: *
Disallow: /

⚠ Only for development / staging sites. Never use on a live site you want in Google.

Common robots.txt mistakes that hurt SEO

Blocking the whole site by accident. A leftover Disallow: / from a staging site is the #1 cause of sudden de-indexing. Always test after uploading.
Blocking CSS and JavaScript. Google renders pages like a browser. If your theme's CSS/JS is disallowed, Google sees a broken page and rankings suffer.
Using robots.txt to hide pages from search results. Disallow stops crawling, not indexing. Use a noindex meta tag for pages you want out of Google.
Putting the file in the wrong place. It only works at the root: yourdomain.com/robots.txt — not in a subfolder.
Forgetting the sitemap line. The Sitemap directive is the single easiest crawlability win and costs one line.
One rule per line, correct casing. Paths are case-sensitive on most servers: /Admin/ and /admin/ are different folders.

Frequently asked questions

What is a robots.txt file?

It's a plain text file in your website's root folder (yoursite.com/robots.txt) that tells search engine crawlers which pages and folders they may or may not crawl. Bots like Googlebot check it before crawling anything else on your site.

Where do I upload my robots.txt file?

In the root directory of your domain, so it loads at yoursite.com/robots.txt. On WordPress, you can use your SEO plugin (Rank Math → File Editor, or Yoast), your hosting file manager, or FTP.

Does Disallow remove a page from Google?

Not reliably. Disallow stops crawling, but a page with external links can still appear in results as a bare URL. To remove a page from the index, let Google crawl it with a noindex tag, or protect it with a password.

What is the best robots.txt for WordPress?

Allow everything except the admin area, keep admin-ajax.php reachable, and link your sitemap: that's exactly what the WordPress preset above generates.

Can I stop ChatGPT and other AI bots from using my content?

You can ask them to stay out by adding groups for GPTBot, ClaudeBot, CCBot and Google-Extended with Disallow: / — the "Block AI crawlers" preset does this in one click. Reputable AI crawlers honour it; it won't stop scrapers that ignore the protocol.

How do I check if my robots.txt is valid?

Generate your file here, run a few URLs through the tester above, then after uploading, verify it in Google Search Console → Settings → robots.txt tester, which shows Google's own view of your file.

Do I even need a robots.txt file?

Small sites can rank fine without one, but having a correct file avoids 404s in crawler logs, protects your crawl budget, and gives you control the day you need it. Every site larger than a handful of pages benefits.