How robots.txt works
robots.txt sits in the root of your site and tells bots which paths they should not crawl. It is a request, not a lock: private pages still need a password.
Good to know
- Upload the file as robots.txt in the root folder (public_html)
- Do not block CSS and JavaScript files that pages need
- Blocking a page in robots.txt does not remove it from Google; use noindex for that
- AI crawler blocks stop new training crawls but not content already collected