robots.txt generator for AI crawlers
Fetch what you have, decide which AI crawlers may read the site, and copy a clean file. Blocking a crawler is a business decision — this makes it a visible one.
No sign-up← All tools
AI crawlers
0 known# Generated by AudiFolio — review before publishing. # Allows the AI crawlers listed below; everything else follows the default group. User-agent: * Allow: /