Robots.txt and XML sitemaps explained, with examples

Two small text files decide a lot about how search engines see your website: robots.txt tells crawlers where they may go, and sitemap.xml tells them which pages you want found. Both are easy to set up, and both are easy to get wrong in a way that quietly costs you traffic.

What robots.txt does (and what it does not do)

Robots.txt lives at the root of your domain, for example https://example.com/robots.txt. Before a well-behaved crawler such as Googlebot fetches a page, it reads this file and follows its rules.

• It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other sites link to it. To keep a page out of search results, use a noindex meta tag and leave the page crawlable so Google can see that tag.
• It is a request, not a lock. Bad bots ignore it. Private pages need a password, not a Disallow line.
• Each subdomain needs its own file. shop.example.com does not use the rules from example.com.

A safe robots.txt for most websites

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

For WordPress, the usual pattern is to block the admin area but keep the AJAX endpoint open, because themes and plugins load content through it:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

You can build either version in a few clicks with our Robots.txt Generator, including optional rules for AI crawlers such as GPTBot.

The robots.txt mistakes that hurt the most

1. Leaving Disallow: / after launch. Developers often block the whole site on a staging server and the rule travels to the live site. One line hides everything.
2. Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your styles and scripts, it may decide the page is broken or not mobile friendly.
3. Using robots.txt to remove pages from Google. As explained above, use noindex instead.
4. Typos in paths. Rules are case sensitive and match from the start of the path, so /Blog and /blog are different.

What an XML sitemap is for

A sitemap is a list of the URLs you want search engines to know about, with optional dates showing when each page last changed. It does not guarantee indexing, but it helps new pages get discovered quickly and helps Google understand which URL is the main version of each page.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/services/</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
</urlset>

Sitemap rules worth following

• List only pages that return status 200, are indexable and are the canonical version. Leave out redirects, 404 pages and pages with noindex.
• Keep each file under 50,000 URLs and 50 MB. Bigger sites use a sitemap index that points to several files.
• Use full, absolute URLs on the same domain as the sitemap.
• Keep lastmod honest. Updating the date on every page every day teaches search engines to ignore it.

Most content systems create a sitemap for you. For a small custom site, our XML Sitemap Generator crawls your pages and builds the file, skipping redirects and noindex pages automatically.

How to check both files in one minute

1. Run the Robots.txt and Sitemap Checker on your homepage. It reads both files, validates the XML and lets you test any path against your rules.
2. Make sure your important pages show as allowed for Googlebot.
3. Submit the sitemap in Google Search Console and Bing Webmaster Tools, then check back after a few days for errors.

Then run a full SEO scan to catch everything else, from titles to page speed.