robots.txt
In short
A plain text file at the site root that says which bot may crawl which URLs.
Example
A "Disallow: /" line under "User-agent: GPTBot" keeps OpenAI's training bot away from the whole site.
Where you'll see it in AILens
The robots.txt health (TEC-001), AI bot access (AGT-010) and training bot decision (AGT-011) controls.
robots.txt is a request, not a lock. And even when you allow a bot, a firewall can still block it; AGT-012 checks for that.
Often confused with
Check what this article covers on your own site.
Check your site