How the Federal Regulatory Data Hub enforces per-client and per-tier rate limits at 8,000 req/s without a centralized counter store: a five-tier quota table (free/researcher/compliance/vendor/internal), token-bucket burst enforcement in Cloudflare KV with ETag-based conditional writes and fail-open after three race retries, and sliding 24-hour window daily quota counting using per-minute KV buckets with a short-lived summary cache for the common below-quota path.
April 2025
7 dated technical articles. These pages record the methods and claims as published; follow each article's source links and update notes for its current scope.
How the Federal Regulatory Data Hub serves 50M+ records via Cloudflare Workers: 8 vertical D1 shards by agency group, Promise.all fan-out for cross-agency queries, entity bridge join across CIK/UEI/LEI/DUNS/NPI, FTS5 full-text search for narrative datasets, response caching with TTL table by endpoint type, and p50/p99 latency budget including partial-response fallback when a shard is unavailable.
How the Federal Regulatory Data Hub implements bitemporal versioning across 50M+ regulatory records in Cloudflare D1: the valid_from/valid_until row-version pattern using half-open intervals, an append-only record_versions audit table with before-state JSON payloads, AS-OF query rewriting in the Workers router using the idx_sdn_pit covering index for sub-5ms p99, three screening modes (current/as-of/historical), and keyset-paginated NDJSON snapshot export for retroactive batch compliance screening.
How the Federal Regulatory Data Hub monitors the freshness of 208 federal datasets and alerts on staleness: per-source FRESHNESS_CONFIG with expected_cadence and max_staleness_hours, D1 dataset_ingests staleness query, Cloudflare Cron */5 * * * * staleness check, multi-channel alerting (Slack webhook, email, PagerDuty) with KV deduplication, OFAC ETag monitoring with 90-minute publish-delay alert, five ingest error classes, and public /status endpoint.
How Voidly selects and distributes its probe vantage network: why ASN diversity matters more than geographic spread, the operator safety constraints that shape high-risk country probes, and how we reach countries where most people connect on mobile-only networks.
Censorship and information control · Engineering and infrastructure
How Voidly delivers measurement configuration to probes without a persistent control channel: gzip+CBOR bundles signed with Ed25519 (signature verified before decompression to prevent zip-bomb attacks), a pull-based auto-update scheduler with 6-hour intervals and exponential backoff, version pinning and two-snapshot rollback, and anonymous country tokens derived via BLAKE3 from ISO code + epoch-week salt so the CDN cannot correlate which overlay a probe applies.
Censorship and information control · Engineering and infrastructure · Cybersecurity and privacy
How Voidly protects probe operator identity while publishing full measurement data: probe_id derived as SHA-256(public_key_bytes) with zero IP logging, human-readable codename system (450K+ combinations, no joint table with probe_id), measurement anonymization (probe_cc + probe_asn published; IP never stored), per-probe Ed25519 signing with isolated key store, and 12-country extra protections (4–48 hour publication delay, 90-day probe_id rotation).
Censorship and information control · Engineering and infrastructure