AI crawler collection

Collection availability

The Crawlers report can display saved activity when available. Site-scoped write keys and an ingestion endpoint must be completed before a connector can be offered.

A future installation needs an approved connector and owner approval for any change to live traffic. Local fixtures do not establish real collection, production overhead or a 48-hour analytics comparison.

What visits can tell you

Search, training and user-requested retrieval serve different purposes. A claimed user agent is unverified unless its request IP matches published provider ranges. Range membership does not independently establish purpose. Crawling does not prove citation; direct citations come separately from saved answers.

Reports show available UTC daily aggregates; collection completeness is unknown. Only pages with a recorded crawl are included; cited pages with no recorded visits are unavailable. Unknown citation evidence stays unknown. Not connected is separate from zero activity. A robots finding needs actual policy evidence and a complete collection window.

Proposed collection boundary

The proposed request fields are timestamp, path, HTTP status, user agent and transient IP for provider-range verification. Query strings are excluded. A future ingestion service must discard request IPs before persistence. Collection is unavailable today, so this page provides no setup or installation action.