Build a valid robots.txt file in under a minute — no coding needed. Choose a preset, allow or block the bots you want, add your sitemaps, then copy or download your finished file.
How to use
- Pick a preset for WordPress, WooCommerce or general SEO, or start from a blank file.
- Allow or block Googlebot, Bingbot and AI crawlers, and add your sitemap URLs.
- Copy your robots.txt or download it, then upload it to the root of your website.
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a small text file in your site’s root directory that tells search engine crawlers which pages and folders they may or may not crawl.
Will robots.txt hide a page from Google?
Not reliably. Robots.txt only asks crawlers not to crawl a page — it does not stop Google indexing a URL it already knows about. Use a noindex tag to keep a page out of search results.
Which bots should I block?
Block bots that waste your crawl budget or scrape your content, such as aggressive SEO crawlers. Be careful with AI crawlers: blocking them may also stop your content appearing in AI search results.
Learn more in our guide: How to Opt Out of AI Training with Robots.txt. Or go back to the Free SEO Tools page.