Indexing: what it is, how Google decides what gets indexed and how you can influence it

Definition

Indexing – or search engine indexing – is the process by which search engines store and organise content they have crawled, so that pages can be shown in search results and used in AI-generated answers.

Indexing is the step that comes after crawling (see our explanation of crawlability). Once Google or another search engine has crawled a web page, it decides whether to save the content in its index – the database used to answer search queries.

Just because a page can be crawled does not mean it will be indexed. Search engines are selective, based on signals such as content quality, duplication, internal linking, technical structure and how important the page seems relative to the rest of the website. Pages that come across as thin, isolated or redundant may never be indexed, or may drop out of the index over time.

For SEO, indexing is a basic precondition for visibility. A page that is not indexed cannot rank. In practice, many visibility problems are not ranking problems but indexing problems – especially on larger websites or in programmatic SEO.

For AEO, indexing matters at least as much. AI systems built on Retrieval Augmented Generation (RAG) can only use content that is available in search indexes or in other sources they retrieve information from. If a page is not indexed, it is effectively invisible to traditional search and AI answers alike, however good the content.

So the goal is not just to "be included" but to signal that the content is worth storing, updating and reusing. A clear information structure, strong internal linking, technical quality and high-quality content increase the likelihood of pages being indexed – and staying indexed over time.

Two different problems that are often confused

Google Search Console shows two statuses that are easily mixed up but often call for completely different actions:

"Discovered – currently not indexed" means Google knows the page exists but has not yet visited it. It is a crawl budget problem, not a content problem. Google has put the page in a queue but has not yet got round to it. The fix is usually not to improve the content but to strengthen the signals that the page is worth a visit: more internal links, a clearer structure and stronger topical authority on the domain.

"Crawled – currently not indexed" means Google has visited the page but judged the content not valuable enough to include. This is a content problem. Technical tweaks will not help – the page needs more depth or more original content, or it has to be merged with something stronger.

Google's Gary Illyes makes the importance of quality very clear in this podcast: "The most important is quality. It's always quality. And I think externally, people don't necessarily want to believe it, but the quality, that's the biggest driver for most of the indexing and crawling decisions that we make."

Indexing is affected by the quality signals of the whole domain

Google does not evaluate each page in isolation. It also looks at the URL pattern and the domain's quality history. If Google's earlier crawls of a website have consistently turned up thin, duplicated or unclear content, it lowers the overall crawl budget for the domain. New URLs end up in the queue – and stay there.

So indexing problems on individual pages can be a symptom of a larger structural problem rather than an isolated technical fault to fix page by page. Improving the website's overall quality signal is therefore a precondition for individual pages to start being indexed regularly.

How search engine indexing works in practice

Google's indexing process can be divided into three steps: crawling, rendering and indexing. Googlebot visits the page and fetches the HTML. Google's systems then render the page with JavaScript – a step that can take several weeks for lower-priority pages. Once the page is rendered, its content, structure and signals are analysed: the topical relevance of the text, the page's internal and external link profile, structured data and page quality signals. Finally, Google decides whether to add the page to the index, with what priority, and which search queries it is relevant for. The process is not linear – Google revisits and re-evaluates indexed pages continuously, and a page can drop out of the index if its relevance or perceived quality declines.

How to request indexing in Google Search Console

The URL Inspection tool in Google Search Console lets you ask Google to crawl and index a specific page. Enter the URL in the search field at the top of Search Console, check the current indexing status and click "Request Indexing". Google does not guarantee that the page will be indexed – the tool places the URL in the crawl queue, but Google decides for itself whether the page is valuable enough. The tool is mainly useful when you publish new content, when you have made important changes, or when a page that should be indexed is not.

What affects whether your website gets indexed?

Whether a page gets indexed depends on several factors working together:

• Technical accessibility – The page must not be blocked by robots.txt or have a noindex tag.
• Internal linking – A page with no internal links from pages that are already indexed is hard for Googlebot to discover.
• Content quality – Thin pages, duplicated content and pages without unique substance may not be indexed.
• Crawl budget – Websites with thousands of pages have a limited crawl budget. Avoid wasting budget on irrelevant URLs.
• Domain quality – Google's historical view of the domain's content quality affects the crawl frequency and indexing priority for new pages.