Robots.txt

Definition

Robots.txt - A file at the root of a website (yourdomain.com/robots.txt) that tells search engine crawlers which pages to access or ignore. Accidental misconfigurations in robots.txt can block an entire site from Google - one of the most common technical SEO mistakes.

In more depth

The robots.txt file is a plain text file that sits at the root of every website and communicates crawling instructions to search engine bots. It uses a simple syntax: User-agent specifies which bot the rule applies to, Disallow blocks specific paths, Allow permits access to paths within a disallowed directory, and Sitemap points to the sitemap location. Most modern SEO tools will crawl your robots.txt as part of their audits.

The most catastrophic robots.txt mistake is blocking the entire site accidentally - typically by a developer adding Disallow: / when working on a staging environment and failing to revert it before launch. This happens more often than you'd expect, and can result in a site being removed from Google's index within weeks. Always check your robots.txt after any major site migration, CMS update, or environment change.

Robots.txt is a directive, not a security measure. Bots that respect robots.txt (Googlebot, Bingbot, most legitimate crawlers) will follow the rules. Malicious scrapers and bad bots generally won't. Don't rely on robots.txt to prevent sensitive content from being accessed - use authentication or server-side access controls for that purpose. Also note: robots.txt can prevent crawling but not indexation - Google can still index a URL it's been told not to crawl if other sites link to it.

Related terms: Crawlability - Indexation - Canonical Tag

Frequently asked questions

Should I block anything in my robots.txt?

Yes, in some cases. Common things to block: admin panels (/admin/, /wp-admin/), internal search results pages (/search?q=), duplicate filtered product pages (/products?color=red&size=large), and staging or development subdirectories. Don't block anything you want Google to index or crawl. Always test robots.txt changes in Google Search Console's robots.txt Tester before deploying.

Does disallowing a page in robots.txt remove it from Google?

No - disallowing crawling in robots.txt prevents Googlebot from accessing the page, but if other sites link to that URL, Google can still index it without the content. To prevent a page from appearing in search results, use a noindex meta tag or X-Robots-Tag HTTP header on the page itself. These require the page to be crawlable, so don't combine noindex with a robots.txt Disallow on the same URL.