SEO Scan Instant

Robots.txt Generator

Choose what search engines and bots may crawl, then download your robots.txt.

Block AI crawlers
Block SEO tool crawlers

robots.txt


    

Upload the file as robots.txt to the root folder of your site, then test it with our Robots.txt Checker.

How robots.txt works

robots.txt sits in the root of your site and tells bots which paths they should not crawl. It is a request, not a lock: private pages still need a password.

Good to know

  • Upload the file as robots.txt in the root folder (public_html)
  • Do not block CSS and JavaScript files that pages need
  • Blocking a page in robots.txt does not remove it from Google; use noindex for that
  • AI crawler blocks stop new training crawls but not content already collected