How to let AI crawlers read your site

The two jobs are not the same

AI companies run different crawlers for different purposes, and lumping them together is what causes most of the confusion. One kind fetches your page at the moment a user asks a question, so that the assistant can answer with current information and credit the source. That crawler can send you a visitor.

The other kind collects pages to train future models. It gives you nothing back directly. Blocking it is a perfectly reasonable business decision.

Because they are named separately in robots.txt, you do not have to make one decision for both.

A sensible default

For most businesses that want customers, the sensible default is to allow the citation and search crawlers and to make your own decision about training crawlers based on how you feel about your content being used that way.

Publishers with valuable archives often refuse training while allowing citation. Service businesses usually allow both, because the exposure is worth more to them than the content is.

Check what you are actually blocking

Only an explicit disallow rule for a named agent blocks it. A missing rule is permission, not refusal. Many sites believe they are blocking AI crawlers when their robots.txt does nothing of the sort, and many others block by accident through a copied snippet.

It is worth reading your own robots.txt rather than assuming. If the file cannot be fetched at all, no conclusion can be drawn safely, and a tool that reports a definite answer in that situation is guessing.

A user agent is a claim, not an identity

Anything can call itself GPTBot. Scrapers do it routinely to look legitimate. So a log line saying GPTBot visited is evidence that something presenting that name visited, which is not the same as proof.

Treat crawler logs as directional. If you need certainty, the major providers publish official address ranges you can verify against, but for most decisions the claimed name is enough as long as you know what it is worth.

What Sondavo does here

Sondavo reads your robots.txt and reports, agent by agent, which AI crawlers you allow and which you block, treating only an explicit refusal as a block. If your robots.txt cannot be fetched, we say we could not check rather than reporting a pass or a fail.

You can also paste a server access log and see which AI crawlers actually visited which pages. That runs entirely in your browser, so the log never leaves your machine, and the results are labelled as agents presented rather than agents verified.