A deep dive into the DNS layer of Voidly's censorship detection: dual-resolver design (ISP resolver vs. neutral control), four interference types (NXDOMAIN injection, IP spoofing, empty answer, timeout), the compare_dns_results() algorithm, known injection IP database (China 18 IPs, Iran 3, Turkey 2), CDN geofencing false positive mitigation via ASN group matching, DNSSEC validation limitations, and DoH/DoT diagnostic queries.
August 2025
6 dated technical articles. These pages record the methods and claims as published; follow each article's source links and update notes for its current scope.
How Voidly publishes its measurement corpus to external researchers: a keyset-paginated NDJSON streaming API with (ts, measurement_id) cursor and Server-Sent Events mode, nightly PyArrow Parquet generation sorted by (domain, ts) for 60% I/O reduction on single-domain queries with zstd level-3 compression, atomic HuggingFace Dataset Hub push with dataset card regeneration, and classifier_version tagging to keep probability distributions comparable across model updates.
Censorship and information control · Transparency and open data · Engineering and infrastructure
A complete field-by-field guide to the Voidly CC BY 4.0 measurement dataset — probe identity, DNS/TCP/TLS/HTTP layers, control comparison, ML classification output, BGP signals, corroboration fields, and filtering recipes for journalists and ML researchers.
Censorship and information control · Engineering and infrastructure · Transparency and open data
Voidly's TimescaleDB continuous aggregates: pre-aggregating 2.2B probe measurements for fast queries
The three-level TimescaleDB continuous aggregate hierarchy behind Voidly's sub-10ms query latency: measurement_hourly (15-minute refresh), country_daily_summary (1-hour refresh), and country_monthly_stats (daily), cutting a 7-day country query from 4.1 seconds to 4ms. Covers refresh policy configuration, late-arriving probe data handling (94.2% within 1 hour, 98.7% within 24h), compression interplay after 7 days, asn_hourly_summary design, and manual backfill procedures.
Censorship and information control · Engineering and infrastructure
The full path from raw probe bytes to a queryable TimescaleDB record: protobuf over QUIC, Cloudflare Worker validation, Kafka fan-out, Rust normalization, probe-version schema drift handling, quality filtering (3.2% drop rate), and nightly Parquet export to HuggingFace.
Censorship and information control · Engineering and infrastructure
How Voidly ingests BGP data from RIPE NCC RIS, RouteViews, and bgp.tools: MRT format parsing, per-country baseline computation, withdrawal detection thresholds, BgpEvent records in TimescaleDB, and how bgp_outage_score is attached to probe measurements.
Censorship and information control · Engineering and infrastructure