CrashedLinkBot

CrashedLinkBot is the crawler behind the CrashedLink broken-link ranking. It reads a small, fixed slice of a website and checks whether the links on it still work. This page explains exactly what it does and how to control it.

How it identifies itself

Requests carry one of these user agents:

CrashedLinkBot/2.0 (+https://crashedlink.com/bot)
Mozilla/5.0 (...) Chrome/128.0.0.0 Safari/537.36 CrashedLinkBot/2.0

The second form is used to check links, because many sites answer a plain bot user agent differently from a browser. Both always contain the token CrashedLinkBot. Requests come from Google Cloud (us-central1); there is no fixed IP range yet, so identify the bot by its user agent.

What it does

• Opens the homepage and follows internal links two levels deep: at most 100 pages per site.
• Checks up to 1,500 unique links found on those pages with GET requests, reading only the status.
• Never submits forms, never logs in, never runs JavaScript, never downloads files.
• Paces itself to about five requests per second per host, two at a time, and slows down as soon as a host answers 429 or 403. If a host rate-limits it, it pauses and re-checks slowly.
• Confirms every 404 a second time before listing it, and re-checks suspicious ones minutes later.
• Visits each site at most once every 7 days, or when the owner of its full report asks for a rescan.

The full rules and the score formula are on the methodology page.

Controlling it with robots.txt

CrashedLinkBot respects robots.txt: rules for CrashedLinkBot or *, both Disallow and Allow. To keep it out of a section or the whole site:

User-agent: CrashedLinkBot
Disallow: /private/

User-agent: CrashedLinkBot
Disallow: /

Blocking the crawler does not remove a site from the ranking; it makes the site unscorable, and its page says so. To be removed, use the address below.

Opting out, disputing a link, questions

• Remove a site from the ranking: write to hello@toob.com.br from an address at that domain, with the domain in the subject. We stop scanning it and take its page down.
• Dispute a broken link: send the URL and the page it was found on. We re-check it from a second network and fix the listing if it works.
• Anything else: same address. A person reads it.