What IETF RFC 9309 - Robots Exclusion Protocol (Standards Track, September 2022) requires
IETF RFC 9309 standardises the Robots Exclusion Protocol as an IETF Standards Track document published in September 2022, formalising the de facto robots.txt convention used since 1994…
This is a preview. The full record adds the complete summary, 8 workflow steps, 12 schema fields and 8 primary citations. Start a free trial or see pricing.
Pillar: AI Governance & Law · Authority: Internet Engineering Task Force (IETF) - RFC 9309 Robots Exclusion Protocol, Standards Track, September 2022 · Version: 1.0.0 · Last updated:
Primary source: https://www.rfc-editor.org/rfc/rfc9309
SHA-256 integrity: b33cb67a5b77a9821bd35bedd4ee06280945815bc0f0c514d4f98b547a9cabc7
Primary Citations — 8 traced to source
- IETF RFC 9309, Robots Exclusion Protocol, by Martijn Koster, Gary Illyes, Henner Zeller, and Lizzi Sassman, Standards Track, September 2022
- RFC 9309 location requirement: the robots.txt file is located at the root of the host and is fetched by clients before crawling
+ 6 more citations (full bibliography, deterministic workflow, actionable schema and crosswalks) included in the vault unlock — $0.01 via Skyfire / L402 / Direct Base USDC.
Access
- Discovery (free): /api/v1/nodes/ietf-rfc-9309-robots-exclusion-protocol-2022.json — 6-field metadata
- Vault (full node): /api/v1/vault/nodes/ietf-rfc-9309-robots-exclusion-protocol-2022.json — full 13-key payload, $0.01 USDC (L402/Skyfire/Direct Base)
- Canonical URL: https://bidda.com/intelligence/ietf-rfc-9309-robots-exclusion-protocol-2022
- Back to registry: Browse all 10,102 compliance nodes