Robots.txt is a simple text file that tells search engine crawlers which parts of your website to access and which to skip. Despite its simplicity, it is one of the most commonly misused technical SEO files. Incorrect robots.txt rules can accidentally block Googlebot from crawling important pages, prevent CSS and JavaScript from being indexed (breaking how Google renders your site), or exclude canonical pages from crawling while leaving duplicate versions accessible. Understanding how to write and maintain robots.txt correctly is basic technical SEO hygiene that every website owner should have.
What Is Robots.txt and How Does It Work?
Robots.txt is a plain text file located at the root of your domain (yourdomain.com/robots.txt). It uses a simple syntax to specify which user agents (bots) can and cannot access specific areas of your site.
A typical robots.txt file looks like this:
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xml
The User-agent: * instruction applies to all crawlers. You can also target specific bots: User-agent: Googlebot applies only to Google's crawler.
Disallow: followed by a path tells the bot it should not crawl that path. Allow: followed by a path explicitly permits crawling, which is useful when you have a broad Disallow rule and need to create exceptions.
Critically: robots.txt controls crawling, not indexing. A page blocked in robots.txt cannot be crawled, but it can still be indexed if Google discovers it through a link from another site. If you want a page not to appear in Google's index, you need a noindex meta tag on the page, not a robots.txt block. This is a fundamental distinction that many website owners confuse.
What Should You Block with Robots.txt?
The pages and directories appropriate for robots.txt blocking are those that:
- Contain admin functionality not intended for public access
- Create duplicate content that you do not want crawled (and are handling with canonical tags for indexing purposes)
- Are parameter-based pages with no unique content value
- Waste crawl budget without contributing to organic visibility
Common appropriate blocks:
CMS admin areas: /wp-admin/, /administrator/, /admin/, and similar backend paths. There is no reason for Googlebot to access your admin interface.
Internal search results: /search/, /?s= (WordPress search parameter), /search-results/. Internal search results pages create near-infinite URL combinations with no unique content value.
Staging and test areas: /staging/, /test/, /dev/. These areas, if publicly accessible, can create duplicate content and should be blocked from crawling. Better practice is to restrict staging environments via server authentication, but robots.txt is a useful secondary safeguard.
Parameter URLs that create duplicates: URL parameters used for sorting, filtering, session tracking, and analytics (?sort=, ?utm_source=, ?sessionid=) can create duplicate content at different URLs. Blocking these parameters reduces crawl budget waste, though canonical tags handle the indexing dimension separately.
User account pages: /account/, /my-account/, /orders/ and similar authenticated areas that Googlebot cannot meaningfully access.
What Should You Never Block with Robots.txt?
These are the blocks that cause serious SEO damage when applied incorrectly:
CSS and JavaScript files: Googlebot needs access to your site's CSS and JavaScript to render your pages as a user would. If your stylesheet or JavaScript files are blocked, Google may see a broken, unstyled version of your pages, affecting how it interprets and ranks your content. Check that your robots.txt does not block /wp-content/, /assets/, /js/, /css/, or similar directories that contain render-critical resources.
Your main content pages: Service pages, product pages, blog posts, and landing pages must be crawlable. A robots.txt rule that is too broad can accidentally block entire content directories. For example, Disallow: /blog without a trailing slash can block everything on the /blog path rather than just the /blog directory.
Image directories: If you have an image sitemap and want your images indexed in Google Images (which can drive referral traffic for product and visual businesses), your image directories must be crawlable.
Sitemap file: Your sitemap.xml file itself must be accessible. It is counterproductive to block the file that tells Google where your content lives.
Canonical versions of important pages: If you are using canonical tags to consolidate duplicate content, the canonical (preferred) version of the page must be crawlable. Blocking the canonical URL defeats the purpose of the canonical tag, because Google cannot crawl the page you are designating as authoritative.
What Are the Most Common Robots.txt Mistakes?
Blocking the entire site: Disallow: / blocks all crawlers from all pages on your site. This is the most catastrophic robots.txt error and the most common one associated with site migrations, CMS updates, and development mode changes accidentally applied to production. Check your robots.txt immediately if you see a sudden drop in impressions and clicks in Google Search Console.
Missing trailing slashes: Disallow: /private blocks everything beginning with "/private", including a potential page called "/privateer" or "/privacy". Adding a trailing slash (Disallow: /private/) restricts the block to the actual directory.
Using robots.txt to hide sensitive data: Robots.txt is a public file readable by anyone, including malicious actors. Listing your admin URL in robots.txt as a Disallow rule tells potential attackers exactly where your admin interface is. Use server-level authentication to protect sensitive areas, not robots.txt.
Conflicting allow and disallow rules: When robots.txt contains both Allow and Disallow rules for overlapping paths, Google uses the most specific rule. If you have Disallow: /private/ and Allow: /private/public-page/, the Allow rule takes precedence for that specific path. However, complex rule interactions can produce unexpected crawling behaviour. Keep your robots.txt as simple as possible.
Blocking your sitemap or sitemap directory: Sites that host sitemaps in a subdirectory sometimes accidentally include that subdirectory in a broad Disallow rule.
How Do You Test and Validate Your Robots.txt?
Google Search Console robots.txt tester: In GSC, open Settings and find the robots.txt tester. This tool shows you which rules apply to specific URLs and whether Googlebot can or cannot access them. Test your most important pages and your admin pages to verify the rules are working as intended.
Fetch and render: Use GSC's URL Inspection tool to fetch and render a page as Googlebot. If Googlebot cannot access CSS or JavaScript, the rendered view will show a broken or unstyled version of the page, revealing whether render-critical resources are being blocked.
Direct browser check: Navigate to yourdomain.com/robots.txt in your browser and read through every rule. Manually verify that the paths being blocked are genuinely directories you intend to exclude, and that no critical content paths are accidentally included.
Coverage report: GSC's Coverage report shows pages that Google has been excluded from crawling due to robots.txt blocks. If unexpected pages appear in this report, investigate immediately.
How Does Robots.txt Interact with Crawl Budget?
Crawl budget is the number of pages Googlebot will crawl on your site within a given period. For most small to medium websites (under 1,000 pages), crawl budget is rarely a limiting factor; Googlebot will crawl your entire site regularly.
For large sites (tens of thousands of pages), crawl budget becomes a meaningful SEO consideration. Blocking parameter-based URLs, paginated archives, and duplicate content variants in robots.txt prevents Googlebot from spending crawl capacity on low-value pages, directing it towards your important content pages instead.
The crawl budget benefit of well-configured robots.txt is most significant for e-commerce sites with faceted navigation, news sites with extensive tag and category archives, and sites with significant legacy URL structures that generate many near-duplicate paths.
Dynamically audits robots.txt configuration as part of every technical SEO review for UK businesses. If you suspect your robots.txt is blocking important content or wasting crawl budget on low-value pages, get in touch for a technical SEO audit.
