Sample robots.txt files

robots.txt is a text file at the root of a site: it tells search crawlers which sections to visit and which to leave alone. Below are ready-made files you can drop in as they are: a plain working one, the defaults of popular CMSs, blocking AI crawlers and scrapers, plus edge cases for testing your own parser

The basics

Sample file: A plain working file: the admin area closed, the sitemap declared

A plain working file: the admin area closed, the sitemap declared basic

273 b

↓

Sample file: Everything open: full access is what an empty Disallow value grants

Everything open: full access is what an empty Disallow value grants allow all

188 b

↓

Sample file: Everything closed: the usual file for a staging site

Everything closed: the usual file for a staging site disallow all

234 b

↓

An empty file, zero bytes empty 0 b

0 b

↓

Defaults of popular CMSs

Sample file: wp-admin closed, with admin-ajax.php allowed back in

wp-admin closed, with admin-ajax.php allowed back in WordPress

318 b

↓

Sample file: The same plus the cart, the checkout and filter links

The same plus the cart, the checkout and filter links WooCommerce

407 b

↓

Sample file: System directories and the Crawl-delay that ships with it

System directories and the Crawl-delay that ships with it Drupal

554 b

↓

Sample file: Cutting off faceted navigation and session parameters

Cutting off faceted navigation and session parameters Magento 2

527 b

↓

Sample file: What the platform serves itself, editable only through a theme

What the platform serves itself, editable only through a theme Shopify

592 b

↓

Sample file: System directories only, the content stays open

System directories only, the content stays open Joomla

438 b

↓

Ready-made for common jobs

Sample file: Closing the site to AI: training and answering a question are different bots

Closing the site to AI: training and answering a question are different bots AI bots 12 agents

834 b

↓

Sample file: Cutting off SEO scanners and link collectors without losing search traffic

Cutting off SEO scanners and link collectors without losing search traffic scrapers

554 b

↓

Sample file: Slowing the crawl: Google ignores the directive, Bing and Yandex do not

Slowing the crawl: Google ignores the directive, Bing and Yandex do not crawl-delay

423 b

↓

Sample file: Different rules for different crawlers, the catch-all group last

Different rules for different crawlers, the catch-all group last multi-agent

635 b

↓

Sample file: Faceted navigation: filters closed, pagination open on purpose

Faceted navigation: filters closed, pagination open on purpose params

767 b

↓

Sample file: Six sitemaps at once, gzipped ones included

Six sitemaps at once, gzipped ones included sitemaps 6 lines

531 b

↓

Edge cases

Sample file: A UTF-8 BOM before the first line, and the rule is lost because of it

A UTF-8 BOM before the first line, and the rule is lost because of it BOM

77 b

↓

Sample file: CRLF line endings, the file saved on Windows

CRLF line endings, the file saved on Windows CRLF

310 b

↓

Sample file: The * and $ wildcards, and which rule wins when several match

The * and $ wildcards, and which rule wins when several match wildcards

622 b

↓

Sample file: Inline comments and the trap of a blank line inside a group

Inline comments and the trap of a blank line inside a group comments

1.1 Kb

↓

Sample file: Deliberately broken: typos, a missing colon, a rule with no group

Deliberately broken: typos, a missing colon, a rule with no group broken

683 b

↓

Sample file: Past the parse limit: 11,089 rules, only the first 500 KiB are read

Past the parse limit: 11,089 rules, only the first 500 KiB are read oversize 512 KiB

520 Kb

↓

🧠 Which rule wins

A crawler obeys exactly one group: the one whose User-agent matches it most closely. It does not also read the group under the asterisk, so rules never add up. Within that group the longest path match wins, and it is the length of the pattern that decides, not the order of the lines: Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ even when it sits below it. On a tie Allow wins. There are only two special characters: * stands for any run of characters and $ pins the end of the URL. And the part everyone forgets: the file blocks crawling, not indexing. A page other sites link to can still turn up in results while being disallowed here, just without a description. To take it out of the index you need an X-Robots-Tag header or a noindex meta tag, and for that the page has to be crawlable in the first place.