Crawler Behavior
Also known as: Bot behavior
The observable choices a search, AI, or other automated crawler makes when requesting and revisiting a website.
Crawler behavior includes which URLs a bot requests, how often it returns, which file types it retrieves, and how it responds after a site changes.
User-agent names can be spoofed. Strong conclusions need additional evidence, and a crawl still does not guarantee indexing or use.