Robots.txt AI Crawler Checker

Review the allow and block status for major AI crawlers, then continue to the full scan if you want the rest of the report.