Robots.txt
Technical SEOA root-level text file that tells crawlers which parts of a site they may or may not access.
Robots.txt uses User-agent blocks with Allow/Disallow rules. Modern versions also govern AI-specific crawlers by name (GPTBot, Google-Extended, etc.), letting site owners separate search indexing permission from AI training permission.