Crawlability – what it is and why it matters

Definition

Crawlability describes how easy it is for search engines to find, reach and crawl the pages on a website.

Crawlability vs indexing – two different things

The two concepts are often mixed up, but they are separate steps:

• Crawlability – can the search engine reach and read the page?
• Indexing – does the search engine choose to store the page in its index?

A page has to be crawlable to be indexed, but crawlable does not automatically mean indexed. Google can crawl a page and still choose not to include it. Crawlability is a necessary but not sufficient condition for visibility.

What affects crawlability?

Good crawlability rests on several factors working together:

• Internal linking – every important page needs several ways in. A page with no internal links at all – an orphan page – is rarely found.
• URL structure – logical, readable addresses make the website easy to understand.
• Site depth – pages that sit many clicks from the home page are crawled less often.
• robots.txt – controls what crawlers may and may not crawl.
• Technical obstacles – broken links, slow responses and incorrect status codes slow down crawling.

Crawl budget – and when it actually matters

Google sets aside a certain amount of resources for crawling each website – a crawl budget. It is governed by two things: how fast the server responds, and how keen Google is to crawl the website.

For most small and medium-sized websites, crawl budget is not a problem – Google gets round to everything anyway. It only becomes relevant on large websites with thousands of pages, especially e-commerce websites with filter and parameter URLs that can generate an almost infinite number of variants. There, the point is to stop crawlers wasting time on pages that do not matter.

Common obstacles to crawlability

The most common reasons pages are not crawled:

• Important pages lack internal links.
• robots.txt blocks pages or resources such as CSS and JavaScript by mistake.
• Long redirect chains that eat crawl budget.
• Content that is only rendered through JavaScript that Google does not execute in full.

Crawlability in AI search

In an AI-driven search environment, crawlability is as fundamental as ever. RAG-based systems can only use content that can actually be fetched. If your pages are not crawlable, they cannot be indexed, cited or used as a source in an AI answer.

Crawlability, in other words, is not advanced fine-tuning but a basic requirement. When a website does not rank, the cause can often be traced back to exactly this – read more about the most common causes in Website not ranking? The 10 most common causes.