Your Robots.txt Might Be Blocking Crawlers (and You’d Never Know)

You check your stats again, and it is the same flat line it was last week. Right about there, a little voice starts telling you the writing must not be good enough, or that you missed some secret everyone else seems to have. Before you listen to that voice, check one small file, because a robots.txt blocking crawlers is one of the quietest reasons a good blog never gets found.

A robots.txt blocking crawlers means the little text file at the root of your site is telling search engines and AI bots not to read your pages. When crawlers are blocked, your posts stay out of Google and never get pulled into answers from ChatGPT, Claude, or Perplexity. You can check yours in about two minutes.

This is not a you problem. I am a former nurse who has spent 17 years in digital marketing, and I have watched brand new bloggers and seasoned site owners get tripped by the very same thing, a single line in a file they never knew to open.

If that turns out to be you, it is one of the faster fixes you will make all month. You can read how I got here on my about page.

Paste your file in below and see exactly who you are letting in and who you are shutting out. Then I will walk you through reading it yourself.

The one file most bloggers never open

Think of robots.txt as the note taped to your front door for visiting robots. Every search engine and AI crawler reads that note before it looks at your site. The note can say come on in, or skip these rooms, or do not come in at all.

What does a robots.txt file do?

Robots.txt is a plain text file at the root of your site that tells crawlers which pages they may read. It does not lock anything, it simply gives instructions that well behaved bots follow. One short Disallow line can wave off Google, Bing, and every AI bot at the same time.

Where do I find my robots.txt file?

Type your domain followed by /robots.txt straight into your browser, like yoursite.com/robots.txt. Whatever loads is the exact note every crawler reads. If you see a page of text, you have one. If nothing loads at all, most crawlers assume they are allowed everywhere on your site.

How to tell if your robots.txt is blocking crawlers

You are hunting for one word and one slash. Open your file and read every line sitting under each User-agent. The line that hides a whole blog is short and easy to skim right past, so a robots.txt blocking crawlers usually goes unnoticed for months.

How do I check if robots.txt is blocking Google?

Open yoursite.com/robots.txt and look for Disallow: / sitting under User-agent: * or under Googlebot. Either one tells Google to stay out. For a second opinion, Google Search Console has a robots.txt report and a URL inspection tool that show whether a page is blocked.

What does “Disallow: /” mean?

Disallow followed by a lone slash means block this entire site. That single slash covers every page, every post, and every image you have published. It is the most common reason a finished blog never shows up, and it is usually left behind by a developer, a theme, or an old staging site.

The WordPress setting everyone blames, and why it is not your robots.txt

Most advice points you straight to one checkbox, and it is only half the story. In WordPress, under Settings then Reading, there is a box called Discourage search engines from indexing this site. People flip it off and expect their robots.txt to change.

Since WordPress 5.3, that box no longer touches robots.txt at all. It adds a noindex tag to your pages instead. So you can have that box unchecked and still be blocked in your robots.txt by a completely separate line.

Does “Discourage search engines” change my robots.txt?

Not anymore. Since WordPress 5.3, the Discourage search engines box adds a noindex tag to your pages rather than editing robots.txt. Still uncheck it if your site is live, but know that a real Disallow line in your robots.txt is a separate problem you fix inside the file itself.

Is your robots.txt blocking AI crawlers?

This is the part almost nobody checks. Google is not the only one reading that file now. ChatGPT, Claude, and Perplexity each send their own crawlers, and if your robots.txt blocks them, your post will never surface in an AI answer no matter how good it is.

Getting this file right is only step one. Whether those engines can actually read and make sense of your pages is a separate question, and it is the one I walk all the way through in Is Your Blog Actually Visible to AI?

Cloudflare looked across its network and found only about 14 percent of top sites give AI crawlers any instructions at all, and GPTBot is the single most blocked bot on the web. Some of that is on purpose. A lot of it is a plugin or a host quietly doing it on your behalf, without ever asking you.

How do I know if I am blocking ChatGPT or Claude?

Look through your robots.txt for the bot names GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. If any of them sits above a Disallow: / line, you are shut out of that tool’s answers. A blanket User-agent: * with Disallow: / blocks all of them at once, even the ones you never named.

How to fix a robots.txt that is blocking crawlers

The fix for a robots.txt blocking crawlers is usually deleting one line, not writing a whole new file. Most bloggers never need to touch code for this, because your SEO plugin can edit the file for you.

How do I fix robots.txt in WordPress?

Open Yoast or Rank Math, find the robots.txt or File Editor tool, and remove any Disallow: / line under User-agent: *. Save, then visit yoursite.com/robots.txt to confirm. If a plugin created the block, that same editor lets you undo it without opening your site’s code.

If you want a clean starting point that lets Google and the AI engines all the way in while keeping your admin area private, this is a safe file for most blogs:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap_index.xml

That opens your whole site to every crawler, search and AI alike, and only holds back the login area. If you ever decide you want AI engines out, you add a named block for GPTBot or ClaudeBot with its own Disallow, and nothing else changes.

Today, open a new tab and go to yoursite.com/robots.txt. Read the whole thing, out loud if you have to. If you spot Disallow: / on its own, or a bot you actually want sitting under a Disallow, that is your one fix for the week.

You do not need a perfect site to get found. You need the door open. Check the file, fix the one line, and let the work you already did finally get seen, by Google and by the AI tools your readers are asking right now.

Be unpolished,
Angela