Crawler
AI & BotsAn automated program that requests and reads pages on the web.
Crawlers range from search-indexing bots (Googlebot, Bingbot) to AI-training and retrieval bots (GPTBot, ClaudeBot, PerplexityBot). Each identifies itself via a User-Agent string, which is how server logs and robots.txt rules distinguish between them.