Give it a URL list, a CSV export or your current sitemap file. Get back a clean sitemap.xml, a sitemap index when it needs splitting, and a matching robots.txt. No 500 URL cap, no signup.
Paste them, drop files in, or both. Everything gets merged and de duplicated.
The site address resolves relative paths and writes the Sitemap: line in robots.txt. Leave it blank and it is worked out from your URLs. The spec ceiling is 50,000 URLs and 50 MB per file, and this splits automatically at whichever comes first.
Search engines want one canonical URL per page. These strip the usual duplicates.
* matches any run of characters. Everything else is matched literally, so a pattern with no * simply has to appear somewhere in the URL. Matching is case insensitive and runs against the whole URL.
Of the three optional tags, only lastmod is worth your effort. Google reads it, but only while it stays honest. A site where every page claims it changed today gets the whole tag ignored.
Google has confirmed it ignores both of these. Bing treats them as hints at best. They are here because some older tools and internal audits still expect them, not because they will help you rank.
| URL pattern | changefreq | priority |
|---|
Only for sites where the language sits in the first folder of the path, like /en/about and /fr/about. URLs that differ only by that folder get grouped and each one lists all of its siblings. Pages with no language folder are left alone. This makes the file considerably bigger, so skip it on a single language site.
If one of your CSV columns holds image URLs, pick it as the image column on the file above and each page will carry its images as image:image entries. Separate several images with a space, a comma, a semicolon or a pipe. Google caps this at 1,000 images per page, so anything past that is trimmed.
robots.txt is a polite request, not a lock. Well behaved crawlers obey it, badly behaved ones read it and carry on. It is also not a way to hide a page from search results, a blocked page can still be listed if other sites link to it. Use a noindex tag for that.
Google ignores Crawl-delay entirely, and it no longer offers a crawl rate setting either, it works the rate out on its own. Bing and Yandex do read this line.
Optional. Only worth it on a large sitemap, and only if you would rather serve the compressed file. If you use it, point the Sitemap: line at the .gz address instead.
Edit it here if you like. Your edits are kept until you press Build again.
Because the ones that crawl need a server to do the crawling, and crawling costs money. That is the whole business model behind the 500 URL free tier, and it is why the desktop crawler everyone recommends costs a few hundred a year the moment your site outgrows a brochure. If you already have the list of URLs, though, none of that expense applies. Building the XML is the cheap half, and this page does the cheap half properly with no ceiling on it.
Google Search Console is the best source and it is free. Open Performance, switch to the Pages tab, set the date range wide and export to CSV, then drop that file here and pick the page column. Every URL Google has actually seen comes across. For a WordPress site, the pages and posts screens export cleanly, and most SEO plugins publish a sitemap already at /sitemap_index.xml that you can save and drop in. Shopify serves /sitemap.xml the same way. If you already have a sitemap and just need it split, pruned or rebuilt with proper dates, that is the fastest path of all.
It tells a search engine which URLs exist and, if your dates are honest, which ones changed recently. It does not make anything rank and it does not force indexing. Its real value shows on big sites, on pages that nothing links to internally, and on a brand new site with no inbound links yet. A twelve page business site with a decent menu gets found perfectly well without one, though there is no harm in having it.
A sitemap may only list URLs on the same host that serves the file, so a sitemap at example.com cannot list shop.example.com pages. It must live at the root if it is to cover the whole site, because a sitemap in a subfolder can only speak for that subfolder. And every URL in it should be one you would be happy to see in search results, meaning no redirects, no 404s, no noindex pages and no duplicates. Listing a page that redirects is a small, common, entirely avoidable waste of crawl budget.
There are two tick boxes here and they do different jobs. The first writes the standard robots.txt groups for the training crawlers named on it, the ones that collect pages to train models. The second covers the search and answer crawlers, which is a separate decision: those are what fetch a page so ChatGPT or Perplexity can answer about it, so blocking them takes your pages out of those answers. Both only work on crawlers that publish a user agent and choose to honour the file, which includes OpenAI, Anthropic, Google's separate AI agent and Common Crawl. Neither does anything about scrapers that ignore robots.txt, and neither removes anything already collected. Treat it as a clearly stated preference rather than a technical barrier, because that is exactly what it is.