robots.txt – what the file does and why it matters for SEO

Definition

robots.txt is a text file used to control how search engine robots may crawl a website by allowing or blocking access to specific URLs or directories.

What robots.txt does

The file gives crawlers instructions before they start crawling. With rules such as Disallow and Allow, you specify which URLs or directories the robots should and should not visit.

Important: robots.txt affects crawling – crawlability – not how the content is ranked. It is a traffic control tool, not a quality or ranking tool.

The most common misunderstanding: robots.txt does not stop indexing

This is where things often go wrong. Blocking a URL in robots.txt stops Google from crawling the page – but not necessarily from indexing it.

If Google finds a blocked URL via external links, it can still end up in the search results, but without a title and description, because the content has never been read. The result is a page that shows up but looks broken. That makes robots.txt the wrong tool for keeping something out of search.

How to actually prevent indexing

If you do not want a page to be indexed, use a noindex tag – in the page's HTML or via an HTTP header. One important consequence: the page must then be crawlable – otherwise Google can never read the noindex instruction.

Blocking a page in robots.txt and setting noindex on it at the same time is therefore a contradiction – Google never sees the noindex tag on a blocked page. It is one of the most common technical SEO mistakes.

What robots.txt is actually used for

• Blocking technical URLs – filters, parameters, internal search results.
• Keeping crawlers away from low-priority content.
• Steering crawl budget towards the pages that actually matter.

The point is not to block a lot, but to help search engines spend their resources in the right place – something that matters most on large websites.

Common and costly mistakes

An incorrect robots.txt is a recurring cause of serious SEO problems:

• Accidentally blocking the whole website – a single Disallow: / can wipe out your visibility.
• Blocking CSS and JavaScript, which stops Google from rendering the page correctly.
• Relying on robots.txt to hide sensitive pages – the file is public and can be read by anyone.

Always check robots.txt after a launch or migration. We describe how technical errors like this affect visibility in Website not showing on Google? The 10 most common causes.