CrashedLinkBot

CrashedLinkBot is the crawler behind the CrashedLink broken-link ranking. It reads a small, fixed slice of a website and checks whether the links on it still work. This page explains exactly what it does and how to control it.

How it identifies itself

Requests carry one of these user agents:

CrashedLinkBot/2.0 (+https://crashedlink.com/bot)
Mozilla/5.0 (...) Chrome/128.0.0.0 Safari/537.36 CrashedLinkBot/2.0

The second form is used to check links, because many sites answer a plain bot user agent differently from a browser. Both always contain the token CrashedLinkBot. Requests come from Google Cloud (us-central1); there is no fixed IP range yet, so identify the bot by its user agent.

What it does

  • Produces a diagnostic sample of at most 100 pages, following homepage links up to depth two. Each pass has a 250-second network deadline.
  • Discovers up to 1,500 unique links. Checks are bounded by the request and time budgets; undiscovered or unchecked URLs are not assumed healthy.
  • The standard crawler does not submit forms, log in or execute page JavaScript. It reads bounded HTML and selected image and sitemap resources. Optional browser diagnostics are requested separately through Google PageSpeed Insights.
  • Paces itself to about five requests per second per host, two at a time, and slows down as soon as a host answers 429 or 403. If a host rate-limits it, it pauses and re-checks slowly.
  • Confirms 404/410 responses before listing links as broken. Timeouts, bot walls and server errors are inconclusive evidence, not broken-link verdicts.
  • Uses a seven-day result cache. Signed-in fresh requests have a 10-per-24-hour quota (admins are exempt; rejected requests and failed scans do not count) and a 15-minute per-site interval. Administrators may request a methodology migration through the same admission limits; there is no automatic weekly recrawl.

For ranking, it selects the homepage and up to 19 homepage-linked pages, then up to 200 internal destinations. With at least ten verified internal destinations, a second pass starts at least five seconds after the first finishes. Rescans reuse the stored manifest for 30 days; method or canonical homepage changes start a new baseline. Confirmation is per item; unavailable links or checklist resources do not veto an otherwise eligible site. A rescan may therefore make two bounded passes. Unknown or conflicting readings can retain fresh previous item evidence for up to eight days from its last reliable reading. They never renew that timestamp. A verified result stays ranked, with its measurement date shown, until a newer verified result replaces it. Inconclusive attempts do not renew a previous result.

The full rules and the score formula are on the methodology page.

Controlling it with robots.txt

CrashedLinkBot respects robots.txt: rules for CrashedLinkBot or *, both Disallow and Allow. To keep it out of a section or the whole site:

User-agent: CrashedLinkBot
Disallow: /private/

User-agent: CrashedLinkBot
Disallow: /

Blocking the crawler prevents new verification. An existing certificate retains its original measurement date and stays ranked until a newer verified result replaces it; the latest attempt is labeled inconclusive. Blocking does not delete a public report. To request removal, use the address below.

Opting out, disputing a link, questions

  • Remove a site from the ranking: write to hello@toob.com.br with the domain and evidence that you control it. We review ownership and the removal request before changing the public listing.
  • Dispute a broken link: send the URL, the page it was found on and the report date. We review the evidence and any access restrictions; corrected listings must be supported by the published verification rules.
  • Anything else: same address. A person reads it.