Instant estimate
SEO and AEO

robots.txt

robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may and may not visit.

It is a request, not a lock: well-behaved crawlers follow it, others don't. The format was standardised as RFC 9309 in 2022. Many sites now use it to decide which AI crawlers may read their content for training, separately from search.

To keep a page out of search results, use a noindex tag instead: a page blocked in robots.txt can still be listed if other sites link to it.

How Regular approaches it

In Standard, robots.txt blocks everything until launch and then allows crawling and points to the sitemap. Whether to allow AI training crawlers is the client's choice; we explain the trade-off.

Sources and further reading

  1. Introduction to robots.txt, Google Search Centraldevelopers.google.com
  2. RFC 9309: Robots Exclusion Protocolrfc-editor.org

Planning a website?

Get a price range in a minute with our instant estimate, or book a 15-minute intro call. We build fast, light websites for sustainable and circular businesses, in Astro or Framer, whichever fits your team.

Book a 15 min introInstant estimate

Instant estimate

Do you need a content management system (CMS)?

Do you have a website today with content and assets that needs to be migrated?

Only if you answered yes above.

Can you select the custom functionality you require?

Choose all that apply.

Optional.

Ready? Hit submit and you’ll see your estimate right away!

Your answers are only used to prepare your estimate and reply to you.