DNS Time‑Series Compression Algorithms Comparative Study

In the evolving landscape of network observability and cybersecurity analytics, Domain Name System (DNS) telemetry has emerged as a foundational data stream. Its ubiquity, regularity, and low payload cost make it ideal for understanding real-time system behavior, threat actor patterns, and long-term infrastructure trends. However, the scale of this telemetry—frequently measured in billions of events per day—presents a serious storage and query challenge, particularly when retained in time-series format for historical analysis. To address this, a range of compression algorithms have been adopted and developed to optimize storage and performance in DNS time-series databases. This comparative study examines the effectiveness, characteristics, and trade-offs of various time-series compression techniques as applied specifically to DNS telemetry, providing insights into which methods are most suitable under differing constraints of volume, resolution, and query profile.

DNS time-series data is typically derived from aggregated log entries, such as query counts per second, per domain, per resolver, or per client IP. These metrics are often captured at one-second to one-minute granularity and organized into multidimensional vectors indexed by time, domain labels, query types, response codes, and other categorical fields. This structure exhibits two notable properties that heavily influence compression performance: temporal locality and high cardinality. Temporal locality refers to the high autocorrelation in metric values across short time windows—such as a domain consistently receiving similar QPS in successive seconds—while high cardinality arises from the sheer number of unique domains and clients that must be tracked.

One of the foundational algorithms in time-series compression is Gorilla, developed by Facebook and widely implemented in systems like InfluxDB and Prometheus. Gorilla encoding works exceptionally well for numerical values with minor deltas between timestamps, employing XOR-based compression for floating-point values and delta-of-delta encoding for timestamps. When applied to DNS QPS data per domain, Gorilla achieves excellent compression ratios—often exceeding 90% reduction—especially when query patterns are stable or follow predictable diurnal cycles. However, Gorilla’s effectiveness drops when dealing with more volatile metrics such as entropy scores or dynamic query bursts typical of DGA behavior, where value predictability is low.

An alternative and increasingly popular method is delta-RLE with bit-packing, used extensively in columnar data formats like Apache Parquet and adopted by systems like Apache Druid for real-time metric storage. This method combines run-length encoding of small deltas with bit-packing to minimize the number of bits required per value. In DNS applications, where large swaths of domains may see unchanged query volume for many consecutive periods, delta-RLE is highly efficient. It outperforms Gorilla in datasets where zero-change values dominate—such as monitoring 100,000 domains where only 2% are active at any time—by collapsing runs of identical metrics into minimal binary footprints. However, its performance degrades in high-volatility streams with short run lengths.

Frame of Reference (FOR) encoding is another approach, relying on the concept of normalizing all values in a frame (e.g., a one-minute block) against a common base value. It then stores only the deltas, which are often small and easily compressible. DNS telemetry benefits from FOR encoding particularly when query patterns follow slow-moving baselines—such as TTL histograms or per-resolver cache hit ratios that evolve slowly over time. When paired with bit-packing and dictionary encoding for categorical labels, FOR enables dense storage of both numeric and dimension-heavy DNS telemetry.

Time-series databases like TimescaleDB and QuestDB implement hybrid models combining delta encoding, Gorilla-style floating-point compression, and dictionary encoding for categorical dimensions. These hybrids adapt dynamically to the observed data and allow for fast decompression during queries. DNS data stored in TimescaleDB, for instance, benefits from aggressive compression policies where per-tenant telemetry is rolled up and block compressed with heuristics that detect entropy shifts and apply column-specific codecs. Comparative benchmarks have shown that these adaptive hybrids often achieve 5–15% better compression than any single algorithm alone, especially on mixed telemetry workloads.

For edge-deployed scenarios or resource-constrained environments, lightweight codecs such as Run-Length Encoding (RLE) and Simple8b provide a reasonable trade-off between CPU usage and storage efficiency. While these codecs are less sophisticated, they shine in scenarios where DNS telemetry is being transmitted over constrained links or stored in ephemeral caches before batch processing. RLE, in particular, works well for binary flags or sparse metrics—such as whether a given domain matched a threat list in a given window.

Another dimension of compression performance is write amplification and decompression speed, both of which are critical in real-time analytics pipelines. Algorithms like Gorilla offer fast decompression and streamable read capabilities, making them suitable for alerting systems and sliding-window anomaly detectors. In contrast, some dictionary-heavy encodings may require full dictionary hydration before queries can be served, introducing latency during exploratory analysis. Therefore, while dictionary encoding can dramatically reduce the size of per-domain labels—especially when combined with normalization of TLDs, SLDs, and common prefixes—it may add overhead unless carefully managed.

When compression ratios are compared across DNS-specific datasets, results vary significantly depending on the resolution, cardinality, and volatility of the underlying telemetry. In a representative benchmark using one month of passive DNS telemetry from a national resolver, Gorilla achieved an average compression ratio of 12:1 on query-per-domain metrics, while delta-RLE achieved 18:1 due to the preponderance of low-variance domains. FOR encoding yielded a 14:1 ratio but required more CPU for both ingestion and query. For mixed-field datasets including categorical labels and metadata tags, hybrid codecs with dictionary support reduced storage by up to 93%, especially when storing per-record metadata such as resolver ID, response code, and enrichment score alongside time-series metrics.

Ultimately, the choice of time-series compression algorithm for DNS data must consider the specific access patterns and latency constraints of the organization. Systems focused on real-time threat detection may prioritize decompression speed and stream-friendly formats, while compliance-driven archiving may optimize for maximal compression and long-term retention. Hybrid approaches offer the most flexibility, especially when paired with a telemetry routing layer that pre-classifies data by volatility and directs it to the most appropriate codec and storage tier.

The future of DNS time-series compression will likely involve adaptive, machine-learning-driven codec selection, where telemetry engines dynamically profile the entropy and pattern behavior of metrics and choose the best compression strategy on the fly. Additionally, with the rise of approximate query techniques and lossy compression models for large-scale behavior tracking, the landscape may shift toward value retention models where exact per-domain counts are sacrificed for summarized insights, further transforming how DNS telemetry is stored and leveraged.

In conclusion, DNS time-series compression is a critical enabler of scalable observability, detection, and compliance architectures. A detailed understanding of the comparative strengths and trade-offs of available compression algorithms ensures that organizations can store vast quantities of DNS telemetry without compromising on speed, precision, or cost-effectiveness. As DNS continues to play a central role in internet telemetry, the engineering decisions around its storage will remain a pivotal factor in the operational agility of security and analytics platforms.

In the evolving landscape of network observability and cybersecurity analytics, Domain Name System (DNS) telemetry has emerged as a foundational data stream. Its ubiquity, regularity, and low payload cost make it ideal for understanding real-time system behavior, threat actor patterns, and long-term infrastructure trends. However, the scale of this telemetry—frequently measured in billions of events…