Differences and Implementation of Data Deduplication and Compression in Primary Storage and Flash Systems

Data deduplication and compression are two popular techniques for saving storage space; deduplication uses hash algorithms to identify duplicate data blocks, while compression employs byte‑level encoding such as Huffman to reduce data size.

From a results perspective, deduplication can be seen as block‑level compression and compression as byte‑level deduplication; in practice they are often combined, with compression applied after deduplication to achieve additive reduction.

Primary storage vs. backup scenarios

Although deduplication originated in backup systems, its adoption in primary storage introduces several differences:

IO size : backup workloads typically handle megabyte‑scale sequential streams, whereas primary storage (e.g., VDI) deals with tens of kilobytes per IO.

IO pattern : backup is mostly sequential read/write with little overwrite, while primary storage sees a high proportion of random reads/writes and about 90% overwrite in VDI workloads.

Performance goals : backup emphasizes high bandwidth and tolerates latency; primary storage demands high IOPS and low latency, so inserting deduplication can increase response time.

Feature priority : deduplication is mandatory in backup but optional in primary storage, leading to less allocated CPU and memory resources for the feature.

These differences affect when deduplication occurs (often as a post‑process in primary storage), the block‑size strategy (fixed small blocks are preferred), and the deduplication method (sampling is common in backup but rarely used in primary storage).

Implementation examples

NetApp FAS series (and EMC VNX/VNX2) support both online and post‑process deduplication/compression. Their workflow includes real‑time 4 KB chunking with hash fingerprinting, optional online compression before writing, and a later idle‑time phase that sorts fingerprints, builds a fingerprint database, and performs duplicate detection and reference‑count updates.

In all‑flash arrays, deduplication becomes a mandatory feature because it reduces write amplification and extends SSD lifespan. EMC XtremIO and Pure Storage illustrate different approaches:

Data is chunked (8 KB in XtremIO) and hashed with a strong SHA‑1 function.

Fingerprints are distributed across nodes for deduplication; new blocks are written, duplicates increase reference counts.

After deduplication, data is cached and compressed before being flushed to disk.

Key characteristics of XtremIO include pre‑write deduplication/compression, strong hash usage, direct placement of deduped blocks without a global mapping layer, and reliance on per‑disk garbage collection.

Pure Storage, by contrast, uses a weak hash followed by byte‑by‑byte comparison to confirm duplicates, a method also employed by HP 3PAR.

Block size varies among vendors (4 KB, 8 KB, 16 KB, or configurable 512 B–32 KB), and metadata is often managed in two stages: LBA‑to‑fingerprint and fingerprint‑to‑physical‑block mappings.

Scale‑out architectures (e.g., XtremIO with dual‑controller clusters) enable linear performance growth, whereas some solutions (Pure Storage) focus on scale‑up within a single controller.

Hardware acceleration for deduplication is limited; HP 3PAR and Skyera have ASICs that offload hash calculation and byte‑wise comparison.

Overall, modern storage systems combine deduplication and compression to dramatically reduce redundant data, improve SSD endurance, and free up capacity, making these techniques essential for both backup and primary storage environments.

For more information, follow the public account by scanning the QR code provided at the end of the original article.