Crawlability

Definition

Crawlability - Whether search engine bots can access, navigate, and read a website's pages. Issues like blocked robots.txt rules, broken JavaScript, or excessive redirects can prevent pages from being indexed.

In more depth

Before Google can rank a page, it must first be able to crawl it - that is, discover, access, and read the page's content. Crawlability issues can prevent perfectly good content from ever appearing in search results, regardless of how well it's optimised. It's a silent problem: your pages look fine in a browser but are invisible to Google.

Common crawlability problems include: robots.txt rules accidentally blocking key pages, JavaScript-rendered content that Googlebot struggles to process, redirect chains longer than 3-4 hops, server errors (5xx responses) on key pages, and pages buried so deep in the site structure that Google's crawl budget runs out before reaching them.

Google Search Console's Coverage report is the primary diagnostic tool - it shows which pages Google has indexed, which it's found but not indexed, and why. The "Crawled - currently not indexed" status often indicates quality or crawl efficiency problems rather than access issues. For large Australian e-commerce or publishing sites, crawl budget management (ensuring Google's limited crawl capacity is directed at your most important pages) becomes a significant technical consideration.

Related terms: Indexation - Robots.txt - Canonical Tag - URL Structure

Frequently asked questions

How do I check if Google can crawl my pages?

Use Google Search Console's URL Inspection tool to check any individual URL - it shows whether Google has indexed it, when it was last crawled, and any issues encountered. The Coverage report gives a site-wide view. You can also use the Fetch as Google feature to see exactly what Googlebot sees when it visits a page.

What is crawl budget?

Crawl budget is the number of pages Googlebot will crawl on your site within a given period. For most small-to-medium sites, crawl budget isn't a limiting factor. For large sites with tens of thousands of pages, it becomes important to ensure Googlebot prioritises your most valuable pages by keeping internal link structure clean and avoiding thin or duplicate content that wastes crawl capacity.