robots.txt
robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may and may not visit.
It is a request, not a lock: well-behaved crawlers follow it, others don't. The format was standardised as RFC 9309 in 2022. Many sites now use it to decide which AI crawlers may read their content for training, separately from search.
To keep a page out of search results, use a noindex tag instead: a page blocked in robots.txt can still be listed if other sites link to it.
How Regular approaches it
In Standard, robots.txt blocks everything until launch and then allows crawling and points to the sitemap. Whether to allow AI training crawlers is the client's choice; we explain the trade-off.
Sources and further reading
- Introduction to robots.txt, Google Search Centraldevelopers.google.com
- RFC 9309: Robots Exclusion Protocolrfc-editor.org