WEBVTT 00:00:00.000 --> 00:00:04.336 Hi, this is the robots dot TXT Tester. It checks whether a 00:00:04.336 --> 00:00:08.672 crawler is allowed to request a page under a specific robots dot 00:00:08.672 --> 00:00:13.008 TXT file. I’ll explain the basics, run a safe example, show the 00:00:13.008 --> 00:00:17.344 exact rule behind each result, and compare a proposed change before it 00:00:17.344 --> 00:00:18.067 goes live. 00:00:18.067 --> 00:00:22.447 A robots dot TXT file is a public set of instructions for 00:00:22.447 --> 00:00:26.828 crawlers. A crawler is software that requests pages, such as Googlebot. This 00:00:26.828 --> 00:00:31.208 tester answers one clear question: under the file you provide, may this 00:00:31.208 --> 00:00:33.033 named crawler request this page? 00:00:33.033 --> 00:00:36.976 Use it before publishing a risky rules change, when an important page 00:00:36.976 --> 00:00:40.919 stops being crawled, when search and AI crawlers need different access, or 00:00:40.919 --> 00:00:44.862 when a staging block hides a noindex instruction. In every case, the 00:00:44.862 --> 00:00:48.805 goal is to connect a result to the exact rule that caused 00:00:48.805 --> 00:00:49.133 it. 00:00:49.133 --> 00:00:53.572 After adding rules and pages, choose the crawlers here. A user agent 00:00:53.572 --> 00:00:58.011 is simply the crawler name used for matching, such as Googlebot or 00:00:58.011 --> 00:01:02.450 GPTBot. Use a preset for a related group, add a custom name 00:01:02.450 --> 00:01:04.300 when needed, then choose “Test.” 00:01:04.300 --> 00:01:08.772 Let’s start with the button labeled “Try an example.” It loads a 00:01:08.772 --> 00:01:13.244 fictional file, two sample paths, and Googlebot, then runs the check automatically. 00:01:13.244 --> 00:01:17.715 Example data makes the walkthrough repeatable and keeps real websites or customer 00:01:17.715 --> 00:01:18.833 information off screen. 00:01:18.833 --> 00:01:22.997 The first rule says every crawler should avoid paths beginning with slash 00:01:22.997 --> 00:01:27.161 search. “Disallow” means do not request a matching path. The next rule 00:01:27.161 --> 00:01:31.324 creates a narrower exception for slash search slash how search works. We’ll 00:01:31.324 --> 00:01:34.100 test that exception and a regular search address. 00:01:34.100 --> 00:01:37.863 The result is a table. Pages run down the left, selected crawlers 00:01:37.863 --> 00:01:41.627 run across the top, and every cell says “allowed” or “blocked.” Here 00:01:41.627 --> 00:01:45.390 one page is allowed and one is blocked. That is a crawling 00:01:45.390 --> 00:01:49.153 result only; it does not tell us whether either page appears in 00:01:49.153 --> 00:01:49.467 search. 00:01:49.467 --> 00:01:53.677 Now open the red “blocked” cell. The evidence panel repeats the page 00:01:53.677 --> 00:01:57.887 and crawler, names the matching user-agent group, and identifies “Disallow: slash search” 00:01:57.887 --> 00:02:02.097 on line two as the winning rule. You do not have to 00:02:02.097 --> 00:02:03.500 guess which line mattered. 00:02:03.500 --> 00:02:07.857 Below the verdict, the tool separates observed evidence from advice. It shows 00:02:07.857 --> 00:02:12.213 the checked URL, the selected crawler, the winning directive, and the applied 00:02:12.213 --> 00:02:16.570 group. If the block is accidental, narrow that rule or add a 00:02:16.570 --> 00:02:20.200 more specific exception, then rerun the same page and crawler. 00:02:20.200 --> 00:02:24.482 Open the green result and the highlighted line changes to “Allow: slash 00:02:24.482 --> 00:02:28.764 search slash how search works.” When more than one rule matches, Google 00:02:28.764 --> 00:02:33.045 uses the most specific matching path. That longer exception wins over the 00:02:33.045 --> 00:02:35.900 broader block, regardless of which rule appears first. 00:02:35.900 --> 00:02:39.909 The button labeled “Try accidental indexing” shows a common trap. A staging 00:02:39.909 --> 00:02:43.917 page is blocked from crawling, so Google cannot fetch it to see 00:02:43.917 --> 00:02:47.926 a noindex instruction on the page. Robots dot TXT can stop a 00:02:47.926 --> 00:02:51.600 crawl while still leaving the address eligible to appear in search. 00:02:51.600 --> 00:02:56.759 For a broader policy review, choose “AI crawler audit.” One click selects 00:02:56.759 --> 00:03:01.918 the built-in AI crawler names, including GPTBot, ClaudeBot, CCBot, PerplexityBot, and Google-Extended. 00:03:01.918 --> 00:03:07.077 Run the test to compare how the same pages are treated for 00:03:07.077 --> 00:03:08.367 every selected name. 00:03:08.367 --> 00:03:12.203 If a crawler is missing, type its user-agent name and choose “Add.” 00:03:12.203 --> 00:03:16.039 The new name becomes a selected chip beside the built-in choices. This 00:03:16.039 --> 00:03:19.876 is useful for an internal crawler or a newly launched bot, but 00:03:19.876 --> 00:03:22.433 confirm the correct name with its operator first. 00:03:22.433 --> 00:03:26.757 Diff mode compares the current file with a proposed replacement. Here the 00:03:26.757 --> 00:03:31.081 proposal removes the useful exception and blocks that path too. After “Test,” 00:03:31.081 --> 00:03:35.405 the summary counts results that would become blocked or allowed, and changed 00:03:35.405 --> 00:03:37.567 table cells are outlined for review. 00:03:37.567 --> 00:03:41.480 The warning says one page-and-crawler result would become blocked and none would 00:03:41.480 --> 00:03:45.393 become allowed. That small count is the review target. Open the changed 00:03:45.393 --> 00:03:49.306 cell, confirm whether the new block is intentional, and do not publish 00:03:49.306 --> 00:03:52.567 until every important change has an owner and a reason. 00:03:52.567 --> 00:03:56.737 “File health” checks syntax and directive problems. This example is clean even 00:03:56.737 --> 00:04:00.907 though one page is blocked, because the valid rule intentionally blocks it. 00:04:00.907 --> 00:04:05.077 A clean file does not mean every important page is crawlable; use 00:04:05.077 --> 00:04:08.900 the table for policy results and the linter for file problems. 00:04:08.900 --> 00:04:12.446 When the result is ready, choose “Download CSV” to keep the page-by-crawler 00:04:12.446 --> 00:04:15.992 table or “Copy share link” to hand the setup to someone else. 00:04:15.992 --> 00:04:19.538 The link stores its state in the address itself rather than saving 00:04:19.538 --> 00:04:23.085 the audit on the site. Use CSV when the list is too 00:04:23.085 --> 00:04:24.267 large for a link. 00:04:24.267 --> 00:04:28.697 The limits are important. Robots dot TXT controls crawling, not indexing, privacy, 00:04:28.697 --> 00:04:33.127 or access. A live fetch cannot reveal an older file Google may 00:04:33.127 --> 00:04:37.557 have cached during an outage, and the test cannot prove a crawler 00:04:37.557 --> 00:04:41.987 will obey the instructions. Use authentication for private systems and Search Console 00:04:41.987 --> 00:04:43.833 when indexing is the question. 00:04:43.833 --> 00:04:48.490 A good next step is simple: record the page, crawler, verdict, and 00:04:48.490 --> 00:04:53.146 winning line. Change the narrowest rule that fixes the intent, then rerun 00:04:53.146 --> 00:04:57.803 this exact case. After publishing, confirm the live file and monitor real 00:04:57.803 --> 00:04:58.967 crawl behavior separately. 00:04:58.967 --> 00:05:04.411 That’s the complete workflow. Test the pages and crawlers that matter, open 00:05:04.411 --> 00:05:09.856 every surprising result, make the narrowest safe change, and rerun the same 00:05:09.856 --> 00:05:15.300 test. Remember that robots dot TXT controls crawling, not indexing or security.