Suede Signal

Articles

Robots.txt Mistakes Local Florida Sites Make

· 5 min read · Jason Colapietro, Suede

One line in robots.txt can remove a business from AI answers for months, and nobody notices because the site still loads fine in a browser. At Suede we look for the same handful of mistakes on every Florida site we audit, and every one of them is fixable in a few minutes once you know what to look for.

Jason Colapietro wrote this guide from those audits. It walks through the six robots.txt and crawler eligibility mistakes the Suede team finds most often on Palm Beach County sites, how Suede Signal detects each one, and the exact fix.

How do crawlers decide whether they may fetch a page?

A crawler reads /robots.txt, finds the group of rules that names its user agent, and applies the most specific matching rule for the URL it wants. If no group names it, it falls back to the User-agent: * group. When an Allow and a Disallow rule match the same URL, the longer pattern wins, and Allow wins a tie.

Suede Signal applies those same precedence rules when it checks Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot against the exact path you audit. The AI Crawler Access lane carries 25 of the 100 points, the joint-heaviest lane, and it runs up to six checks: one per crawler plus a non-empty llms.txt.

Mistake 1: The staging file shipped to production

A developer builds the new site on a staging domain with this file, and it goes live unchanged:

User-agent: *
Disallow: /

On WordPress, the "Discourage search engines from indexing this site" checkbox can produce the same effect after launch. The fix is to replace it with an explicit allow and a sitemap line:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

At Suede, "check robots.txt" is a line on every launch checklist, and it should be on yours. It is the single most common reason a freshly rebuilt site disappears.

Mistake 2: A bot-specific group that drops your other rules

Site owners often add a group for one crawler and assume the wildcard rules still apply to it. They do not. A crawler that finds a group naming it ignores the * group entirely.

User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

That file is fine for GPTBot. But if you later add User-agent: Bingbot with a single Crawl-delay line, Bingbot now has no Disallow rules at all and will crawl /checkout/. Repeat every rule a specific bot needs inside its own group.

Mistake 3: Blocking search bots when you meant to block training

Many Florida businesses want to keep their content out of model training without vanishing from AI search. The crawlers are separate, and blocking the wrong one costs you visibility:

Purpose OpenAI Anthropic Google
Model training GPTBot ClaudeBot Google-Extended token
Search and answers OAI-SearchBot Claude-SearchBot Googlebot

A file that blocks GPTBot and ClaudeBot but leaves OAI-SearchBot, Claude-SearchBot and Googlebot allowed keeps you eligible for answers while opting out of training use by those two crawlers. The longer Suede guide on which AI crawlers to allow covers the full list, including PerplexityBot and the user-triggered fetchers people forget.

Mistake 4: Prefix rules that catch your city pages

Robots rules match from the start of the path, so they catch more than they look like they catch. This is the mistake we see most on local service sites:

User-agent: *
Disallow: /services

The owner meant to hide an old /services index. The rule also blocks /services-west-palm-beach, /services/jupiter-roof-repair and every other URL that starts with those characters. Add an end anchor so the rule targets exactly the one URL you mean:

Disallow: /services$

Wildcards need the same care. Disallow: /*? blocks every URL with a query string, which is fine until your booking platform or service-area filter puts the only crawlable copy of a page behind one.

Mistake 5: A robots.txt that errors or sits behind a challenge

If /robots.txt returns a 5xx error, times out, or answers with a bot challenge page from a firewall, a careful crawler cannot tell what it is allowed to fetch. Google's documentation says a server error on robots.txt can pause crawling of the whole site. Hurricane season makes this worse in Florida: emergency traffic spikes, a host throttles, and a security plugin starts challenging anything that looks automated.

Suede Signal reports this case as "crawler eligibility is unknown" and fails the check, rather than guessing. A 404 is different: no file means no rules, and the audit treats that as allowed. Suede recommends checking the file from outside your network and confirming that your firewall exempts /robots.txt and verified search crawlers.

Mistake 6: Using robots.txt to hide pages from results

Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it, and Google stopped honoring noindex inside robots.txt in 2019. To keep a page out of results, allow it to be crawled and serve a noindex meta tag or X-Robots-Tag header instead.

Who maintains these checks

Suede Signal is built by Suede, the West Palm Beach agentic platform company founded in 2024 by engineer Jason Colapietro. The robots matching in the audit follows Google's documented precedence rules, and a recent update refined how it handles unreachable robots.txt files and rule precedence, the same edge cases covered above. That kind of correctness work is familiar territory: Jason Colapietro has 54 pull requests merged across 46 open-source repositories he does not maintain, including Jest and Backstage.

Jason Colapietro also holds five Semrush Academy certificates, including Local SEO Essentials, and the HubSpot AEO Fundamentals certification, all listed in the Suede training gallery. On October 31, 2026, Suede SEO ranked #1 among Palm Beach County-based companies in The Manifest's Florida AI Deployment directory and top 25 statewide, as noted on the recognition page. Suede ranks among Clutch's Top 30 AI Consultants in Florida and Top 50 AI Consultants nationwide (November 2026). He wrote about verification as a habit in Proof as Infrastructure, one of his five books on Amazon.

Your next step

Open your own /robots.txt right now, then run your most important city or service page through Suede Signal to confirm all five crawlers are allowed on that exact path. If you find a rule you cannot untangle, or the problem turns out to be a firewall or host setting, the Suede technical SEO team in West Palm Beach can fix it as part of an assessment. Read Jason Colapietro's founder page to see the full record behind the work.

How visible is your site to AI engines?

Run the free audit: up to 22 checks, graded in seconds, nothing stored.

Run your free audit

More articles