Short answer: we read the public robots.txt file of 42 California law firm websites on 7 October 2026. None of them blocked ChatGPT's search crawler, OAI-SearchBot, and none blocked all crawlers. If a firm is missing from AI answers, this sample suggests its robots.txt file is rarely the reason, though it takes about two minutes to check.
Why we looked
OpenAI's documentation says a site that blocks OAI-SearchBot will not appear in ChatGPT search results. That made one line in a public text file a plausible, cheap-to-check cause of invisibility, so we checked how common the problem is among California firms.
What we found
- 0 of 42 blocked every crawler.
- 0 of 42 blocked OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot or Google-Extended.
- 1 of 42 blocked a single AI-related crawler, Common Crawl's CCBot, and nothing else.
- 1 of 42 went further and named AI search crawlers to allow them explicitly.
- 8 of 42 published an llms.txt file.
- 3 of 42 had no Sitemap line in robots.txt.
Many files were the default text a WordPress or SEO plugin writes, which says "everyone is welcome" and never mentions AI at all.
What this means for a firm
A clean robots.txt is good news, but it is not an explanation for why your firm does or does not show up in AI answers. Other causes are more likely: pages that are not indexed, details about the firm that differ between sources, or no page that answers the client's question in plain words. The 30-minute check in Is ChatGPT Describing Your Law Firm Correctly? is the better place to start.
Two minutes is still worth spending. Open yourfirm.com/robots.txt, look for a line that disallows everything for all crawlers, and look for any rule naming OAI-SearchBot. Our guide to what Google and OpenAI document explains what each crawler controls. Ask whoever manages your site to make changes; a wrong rule can also block Google.
What we cannot tell you
- It is not a random sample. We found firms through page-one search results across California practice areas and regions, so larger and better-optimized sites are probably overrepresented.
- robots.txt shows only the rules a site publishes. It does not show restrictions applied elsewhere, such as at a firewall, so a clean file does not prove a crawler can reach a site.
- We did not test whether any firm appears in AI answers. We looked at the file, not at outcomes.
- llms.txt: Google's documentation says no special files are needed for AI Overviews or AI Mode. We report how many firms publish one, not whether it helps.
Method
On 7 October 2026 we collected 42 independent California law firm websites, no more than three per practice area in each group, excluding directories and marketing agencies. For each we read the public robots.txt and checked whether /llms.txt returned a real text file. We recorded which AI-related crawlers each file names and how. Four files were re-read in full as a spot check and matched our notes. Firms are not named. Files change, so this is a snapshot.
This article is general information, not legal advice. Last checked: October 2026.
Frequently asked questions
Does a clean robots.txt mean ChatGPT will mention my firm?
No. It removes one possible blocker. Whether a tool mentions a firm depends on much more.
Should my firm block AI crawlers?
That is a business decision. OpenAI describes search and training as separate crawlers, so the two choices can differ. Ask your web manager before changing anything.
Do I need an llms.txt file?
Google says no special files are needed for AI Overviews or AI Mode. Some firms publish one anyway.
