Skip to content
All tools

Robots.txt generator

Build a robots.txt file.

AI crawlers

Blocking Google-Extended does not affect Google Search rankings.

robots.txt is a request, not a lock: well-behaved crawlers follow it, others may not. To keep a page out of Google, use noindex instead of blocking it here.

Made in your browser.

Next step:Meta tag generatorSERP previewSchema markup generatorllms.txt generatorKeyword density

How to make a robots.txt file

  1. Choose whether search engines may crawl your site.
  2. Add folders to keep crawlers out of, one per line.
  3. Add your sitemap URL.
  4. Download the file and upload it to the root of your site.

Blocking AI crawlers

Tick Block AI training crawlers to ask bots such as GPTBot, ClaudeBot and Google-Extended not to use your content. Search engines are not affected.

Will it hide a page from Google?

Not reliably. A blocked page can still appear if other sites link to it. To keep a page out of results, use a noindex tag.