Diagnosis

Crawl Analysis

Large sites need to know how crawlers — including AI bots — actually spend their time, and which pages are never reached at all. This is a long-standing capability for big sites; the addition here is treating AI bot traffic as a first-class part of it.

What it does

• See how crawl attention is actually spent across your site
• AI bot traffic broken out separately
• Find pages that are never reached at all