# Standard crawlers User-agent: * Allow: / # AI crawlers — give them structured priority User-agent: Google-Extended Crawl-delay: 1 Allow: /ai.json Allow: /protocol/ Allow: /rag/ Disallow: /admin/ User-agent: PerplexityBot Crawl-delay: 0.5 Allow: /ai.json Allow: /feed.json Allow: /proof/ # Voice assistants User-agent: YandexSpeechKit Allow: /ai-voice/manifest.json # Blockchain indexers User-agent: TON-Explorer Allow: /ton-site.json Allow: /proof/