Most small business owners never think about robots.txt until something goes wrong. It is a tiny file, often just a few lines long, and it sits quietly at the root of your website telling search engines what they may and may not look at. Get it right and you will never notice it. Get it wrong and your site can vanish from Google overnight.

Here is what the file does, what it does not do, and the handful of mistakes that cause real damage.

What robots.txt actually is

A robots.txt file is a plain text file that lives at the root of your domain. It gives crawlers — Google, Bing and every other bot that visits your site — instructions about which parts of the site they are welcome to crawl and which parts they should leave alone.

It is usually the very first file a crawler requests when it arrives. Before Google reads a single page of your content, it checks your robots.txt for the ground rules. That is why a mistake in this one small file has such an outsized effect.

What it does — and what it does not do

This is the part that trips up almost everyone. Robots.txt controls crawling, not indexing. Those are two different things.

Crawling is Google fetching and reading a page. Indexing is Google deciding to list that page in search results. Blocking a page in robots.txt stops the first one, but it does not reliably stop the second. If other sites link to the blocked page, Google may still index it — it simply lists the page without being able to read what is on it, which usually looks worse than leaving it alone.

Warning

To genuinely keep a page out of search results, use a noindex meta tag on that page — and make sure the page is NOT blocked in robots.txt. Google has to be able to crawl the page to see the noindex instruction. Blocking it in robots.txt hides the very instruction you want Google to follow.

The basic syntax, in plain English

There are only four instructions you are likely to ever need:

  • User-agent — which crawler the following rules apply to. An asterisk means "all crawlers".
  • Disallow — a path crawlers should not request. Leave the value empty to disallow nothing.
  • Allow — an exception carved out inside a path you have disallowed.
  • Sitemap — tells crawlers where your sitemap lives so they can find all of your pages.

Put together, a simple set of rules reads like this:

robots.txttext
        User-agent: *
Disallow: /private/
Allow: /private/public-page.html

That says: every crawler, stay out of the private folder, except for that one page inside it. Rules are read top to bottom and paths are case sensitive, so a trailing slash or a capital letter in the wrong place genuinely matters.

What a typical small business robots.txt looks like

For most small sites the job is simple: let search engines see the public site, keep them out of the admin and checkout areas, and point them at the sitemap.

robots.txttext
        User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/

Sitemap: https://www.yourdomain.com/sitemap.xml

That is genuinely it. Most small business websites need only a handful of lines. If yours runs to dozens of rules, someone has probably been copying and pasting from the internet, and it is worth having a proper look at what those rules are actually blocking.

The most damaging mistake you can make

One stray line can block your entire website from every search engine on earth:

robots.txttext
        User-agent: *
Disallow: /

A single forward slash after Disallow means "the whole site". This is not a rare or exotic error. It happens most often during a website redesign: the staging site is deliberately configured to block crawlers so the unfinished version does not appear in Google, and then that configuration gets pushed live along with everything else.

The symptom is unmistakable — a sudden, total collapse in organic traffic within days of launch. Not a dip, not a seasonal wobble, but a cliff edge. Rankings that took years to build can disappear in a fortnight, and recovery is rarely instant even once the file is fixed.

Tip

Make checking robots.txt the first item on your post-launch checklist. After any new site launch, migration, platform change or server move, open the file and read it before you do anything else. It takes ten seconds and it is the single highest-value check in technical SEO.

Other common mistakes

  • Blocking CSS and JavaScript files. Google renders your pages much like a visitor's browser does. If it cannot fetch your stylesheets and scripts, it sees a broken, half-built version of your site and judges it accordingly — including whether it works on mobile.
  • Using it to hide private or sensitive content. This does not work, and it backfires. The file is publicly readable by anyone who cares to look, so listing a confidential folder in it is effectively publishing a map to the things you wanted hidden. Use passwords or proper access controls instead.
  • Forgetting to update it after a redesign. Old rules pointing at folders that no longer exist are harmless clutter, but old rules blocking sections that have since become important pages are not.

How to check yours

You do not need any special tools. Type your domain name into any browser followed by /robots.txt — so yourdomain.com/robots.txt — and the file will display as plain text on screen. If nothing appears, you probably do not have one, which is usually fine.

Read the file line by line and ask one question of each rule: do I actually want that blocked? Pay particular attention to any Disallow line with nothing after the slash.

Google Search Console also includes a robots.txt report. It shows you the exact version Google has fetched, when it last fetched it, and flags any errors or warnings it found. That is worth checking because what Google sees is occasionally not what you think you uploaded — caching, redirects and staging servers can all get in the way.

Do you even need one?

Strictly speaking, no. If you are happy for everything on your site to be crawled, you do not need a robots.txt file at all. Crawlers that cannot find one simply assume they are free to crawl the whole site.

That said, a simple file that declares your sitemap location is good practice. It helps crawlers find all of your pages efficiently, it costs nothing, and it gives you an obvious place to add a rule later if you ever need one.

The practical takeaway

Robots.txt is a small file with a big blast radius. Keep it short, keep it boring, and remember that it manages crawling rather than what appears in search results. If you want a page kept out of Google, use a noindex tag and leave the page crawlable.

Above all, look at yours after every launch, redesign or migration. Thirty seconds of checking is a great deal cheaper than months spent rebuilding rankings that were never meant to disappear in the first place.