Free tool
What is your robots.txt blocking?
Put in a website address and we fetch the real file, read it the way a crawler does, and tell you plainly who is shut out.
One line in robots.txt can hide an entire site from search, which makes it the cheapest thing to rule out when pages are missing from Google. The checker separates the crawlers that answer questions about your business from the ones that only collect content for training, because blocking those is a different decision.
Free, no account. We fetch the file once and read it. Nothing is stored.
Questions people ask
- How do I check my robots.txt file?
- Type your website address into the box above. We fetch yoursite.com/robots.txt, read every rule group in it, and report which crawlers are allowed and which are blocked. You can also see the file yourself by visiting that address in a browser.
- Is my robots.txt blocking Google?
- It is if a group that applies to Googlebot contains Disallow: / and nothing narrower overrides it. That single line asks Google not to read any page on the site, which over time removes those pages from results. The checker above tells you plainly whether that rule is present.
- Does robots.txt block ChatGPT and other AI assistants?
- Only if you name them. There are two different populations: agents like ChatGPT-User, OAI-SearchBot, PerplexityBot and Claude-User fetch a page to answer somebody's question and can send that person to you, while GPTBot, ClaudeBot, CCBot and Google-Extended collect content for model training and send nothing back. Blocking the first group removes you from AI answers. The checker separates the two so you can make a deliberate choice.
- What should a robots.txt file contain?
- For most sites, very little: allow crawlers everywhere, disallow the handful of paths that should stay private such as admin or checkout pages, and add a Sitemap: line pointing at your sitemap.xml. Robots.txt is a request to stay out of a page, not a way to keep a page secret, so never rely on it for anything confidential.
- Why is my site not showing in Google?
- A blocking robots.txt is one of several possible reasons and the quickest to rule out, which is why this tool exists. If the file is fine, the usual causes are noindex tags on the pages themselves, pages that are not linked from anywhere, or a site that is simply too new to have been crawled yet.
- Does robots.txt affect rankings?
- Not directly. It controls what gets read, not what gets ranked. But a page a crawler cannot read cannot rank at all, so a mistake here costs you everything downstream of it.
Other free tools
- What keywords does my website rank for?See the searches a site already turns up for, free and without an account.
- SERP snippet previewSee how your title and description will look in Google results before you publish.