What IETF RFC 9309 - Robots Exclusion Protocol (Standards Track, September 2022) requires
IETF RFC 9309 standardises the Robots Exclusion Protocol as an IETF Standards Track document published in September 2022, formalising the de facto robots.txt convention used since 1994. The protocol defines how website operators communicate access rules to automated clients (crawlers, scrapers, AI training data collectors) via a robots.txt file at the site root. RFC 9309 specifies the file format, parsing requirements, and rule precedence for the User-Agent and Disallow/Allow directives. The Protocol is voluntary at the protocol level - it expresses publisher intent but technical enforcement depends on the requesting client honouring the file. In AI governance, RFC 9309 has become the foundation for AI training data opt-out signalling: publishers use robots.txt to express that named AI crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot, anthropic-ai, PerplexityBot, etc.) should not crawl their site for training data collection. Companion mechanisms include the IETF AI Preferences Working Group proposed standard (under development) and the W3C Text and Data Mining Reservation Protocol. The CNIL AI development recommendations (April-June 2024), the EU AI Office GPAI Code of Practice (July 2025) copyright commitment, and the OpenAI, Google, Anthropic, and Common Crawl crawler operator policies all reference robots.txt opt-out honour as the baseline AI training data opt-out mechanism.
Pillar: AI Governance & Law · Authority: Internet Engineering Task Force (IETF) - RFC 9309 Robots Exclusion Protocol, Standards Track, September 2022 · Version: 1.0.0 · Last updated:
Primary source: https://www.rfc-editor.org/rfc/rfc9309
SHA-256 integrity: b33cb67a5b77a9821bd35bedd4ee06280945815bc0f0c514d4f98b547a9cabc7
Primary Citations — 8 traced to source
- IETF RFC 9309, Robots Exclusion Protocol, by Martijn Koster, Gary Illyes, Henner Zeller, and Lizzi Sassman, Standards Track, September 2022
- RFC 9309 location requirement: the robots.txt file is located at the root of the host and is fetched by clients before crawling
+ 6 more citations (full bibliography, deterministic workflow, actionable schema and crosswalks) included in the vault unlock — $0.01 via Skyfire / L402 / Direct Base USDC.
Access
- Discovery (free): /api/v1/nodes/ietf-rfc-9309-robots-exclusion-protocol-2022.json — 6-field metadata
- Vault (full node): /api/v1/vault/nodes/ietf-rfc-9309-robots-exclusion-protocol-2022.json — full 13-key payload, $0.01 USDC (L402/Skyfire/Direct Base)
- Canonical URL: https://bidda.com/intelligence/ietf-rfc-9309-robots-exclusion-protocol-2022
- Back to registry: Browse all 10,099 compliance nodes