Complete ZFS Handbook: Theory, Features, Use Cases, and Command Reference

Introduction to ZFS

ZFS is a file system and volume manager originally developed by Sun Microsystems for Solaris. Today, its development is maintained by the OpenZFS project, present in systems such as FreeBSD, Linux (via kernel module), illumos, macOS (unofficially), and others. It combines file system functionality, logical volume management, data protection, and simplified administration into a single integrated solution.

Its architecture, based on copy-on-write and checksums, ensures data integrity even in hardware failure scenarios, while offering features such as instant snapshots, efficient clones, compression, deduplication, RAID-Z, native encryption, and data replication.

The latest stable version of the series is OpenZFS 2.3, which brought significant changes enough to alter practical recommendations that seemed fixed a few years ago — fast dedup, RAID-Z expansion without downtime, and Direct I/O for NVMe, among others.

History and Context

ZFS was announced in 2004 and introduced in Solaris 10 in 2006. It was designed to solve common problems in enterprise storage systems, such as silent data corruption, volume management difficulty, and scalability limitations. After Sun's acquisition by Oracle, ZFS code was closed, leading to the creation of the OpenZFS project from an illumos fork. Since then, the community has unified efforts to maintain and evolve ZFS in an open manner, with cross-platform support and regular release cycles.

Fundamental Concepts

Storage Pools

ZFS introduces the concept of a storage pool (zpool), a logical collection of disks (or partitions) that provides space for datasets. Unlike traditional systems where each file system resides on a fixed partition, in ZFS all datasets share the available space in the pool. This eliminates the need for partition resizing and allows dynamic space allocation.

A pool can be composed of disks in various configurations (each called a vdev — virtual device):

• Single disk: a single disk, no redundancy. Any failure results in total data loss of the vdev.

• Mirror: mirroring, equivalent to RAID-1. Tolerates failure of N-1 disks in the mirror.

• RAID-Z: parity schemes with three variants — RAID-Z1 (1 parity disk, tolerates 1 failure), RAID-Z2 (2 parity disks, tolerates 2 simultaneous failures), and RAID-Z3 (3 parity disks, tolerates 3 simultaneous failures).

• dRAID: distributed vdev with embedded hot spares in the topology, created to drastically accelerate resilver time in large arrays.

• Striping: distribution without parity among vdevs — increases throughput and capacity, but any vdev without its own redundancy that fails brings down the entire pool.

RAID-Z nomenclature is defined by the number of parity disks — there is no formal equivalence with traditional RAID nomenclature beyond Z1/Z2:

Ashift: the irreversible decision at pool creation

A critical decision at pool creation is ashift — the logical sector alignment used internally by ZFS. Defined as log2 of the sector size, this value cannot be changed after pool creation. Modern disks often report a logical sector of 512 bytes for compatibility ("512e") even though they have a real physical sector of 4K — if ZFS detects it incorrectly, the pool suffers a permanent performance penalty with no solution other than recreating everything.

# Confirm the real physical sector size before creating the pool
smartctl -i /dev/sdX | grep -i sector

💡 When in doubt, always use ashift=12 — the cost of "over-provisioning" is negligible, but under-provisioning is irreversible.

Datasets: File Systems, Volumes, and Snapshots

Within a pool, data is organized into datasets, which can be:

• File system: hierarchical file system, mountable in the operating system namespace.

• Volume (zvol): block device that can be used as a virtual disk for VMs, iSCSI, etc.

• Snapshot: instant, read-only copy of a dataset at a specific point in time.

Each dataset has inheritable properties that control its behavior. Besides compression and quotas, there are several others with direct impact on performance and compatibility:

# atime: disabling reduces unnecessary writes on each read
zfs set atime=off vault/documents
zfs set relatime=on vault/documents    # middle ground: updates only if accessed more than 24h ago

# xattr: extended attributes (needed for Samba, SELinux, etc.)
zfs set xattr=sa vault/documents       # 'sa' is more performant than 'on'

# acltype: POSIX ACLs (needed for Samba/NFSv4 with advanced permissions)
zfs set acltype=posixacl vault/documents

# sync: controls synchronous write behavior
zfs set sync=standard vault/db         # default, respects fsync()
zfs set sync=always vault/db-critico   # forces sync even without fsync (safer, slower)
zfs set sync=disabled vault/scratch    # ignores fsync (fast, risk of loss on crash — only disposable data)

# canmount: controls whether the dataset is automatically mounted
zfs set canmount=off vault/templates

# logbias: optimizes for latency or throughput in synchronous writes
zfs set logbias=latency vault/db       # default
zfs set logbias=throughput vault/backup

Copy-on-Write (COW)

Every write operation in ZFS is transactional: data is never overwritten in place. Instead, new blocks are allocated for modifications, and only after the transaction is confirmed is the root pointer of the block tree updated atomically. This mechanism ensures that the file system remains consistent even after abrupt power or kernel failures, completely eliminating the need for traditional consistency checks (fsck).

Checksums and Data Integrity

ZFS stores checksums of all data and metadata blocks, calculated on write and verified on every read. If there is a mismatch (silent corruption caused by bit rot, controller failure, or media error), ZFS detects the problem and, when redundancy is available (mirror, RAID-Z, or dRAID), self-heals automatically.

The supported and currently recommended checksum algorithms include:

zfs set checksum=sha256 vault/documents   # recommended for dedup
zfs set checksum=blake3 vault/documents   # modern faster alternative
zfs set checksum=fletcher4 vault/documents # factory default, no dedup

This verification occurs only during normal reads — to periodically verify the entire pool, there is scrub, covered in the Practical Administration section, and it is the most important and most often neglected maintenance in ZFS deployments.

ZFS Architecture

ZFS is composed of several layers working together:

• ZPL (ZFS POSIX Layer): interface between system calls and ZFS.

• DMU (Data Management Unit): manages objects and blocks transactionally.

• SPA (Storage Pool Allocator): block allocation, RAID-Z/dRAID management, and checksums.

• ZIL (ZFS Intent Log): records synchronous operations for fast recovery.

• ARC (Adaptive Replacement Cache): main read cache, resident in RAM.

• L2ARC: optional secondary cache, typically on SSDs. Since recent OpenZFS versions, it can be persistent across reboots, eliminating cold cache after restart.

• SLOG (Separate Intent Log): dedicated log device for ZIL (NVMe/Optane SSD), accelerates synchronous writes.

Advanced Features

Snapshots and Clones

Snapshots are created in constant time and initially consume no additional space — they reference only the blocks already existing at the time of creation. As data is modified, old blocks remain retained by the snapshot, while the active dataset references new blocks.

Clones are writable datasets created from a snapshot, sharing blocks with the origin until modifications occur — useful for test environments, selective recovery, or rapid VM provisioning.

The full snapshot management lifecycle, in practice, involves more than just creation:

# List snapshots
zfs list -t snapshot

# Create snapshot
zfs snapshot vault/documents@backup-20260623

# Create writable clone from a snapshot
zfs clone vault/documents@backup-20260623 vault/documents-teste

# Rollback (DESTRUCTIVE — discards changes after the snapshot)
zfs rollback vault/documents@backup-20260623

# Forced rollback, destroying newer intermediate snapshots
zfs rollback -r vault/documents@backup-20260623

# "Promote" — reverses dependency between clone and origin
# (useful when the clone becomes more relevant than the original dataset)
zfs promote vault/documents-teste

# Destroy snapshot
zfs destroy vault/documents@backup-20260623

# Destroy all snapshots of a dataset (careful!)
zfs destroy vault/documents@%

Automatic snapshots with retention routine:

zfs snapshot vault/documents@hourly-$(date +%Y%m%d-%H%M)
zfs snapshot vault/documents@daily-$(date +%Y%m%d)
zfs snapshot vault/documents@weekly-$(date +%Y-W%V)

# Keep only the last 7 daily snapshots, discarding the rest
zfs list -t snapshot -o name -S creation | grep 'daily-' | tail -n +8 | xargs -n1 zfs destroy

Compression and Deduplication

ZFS offers transparent compression, applied automatically without intervention from applications. Available algorithms include lzjb, gzip (levels 1–9), zle, lz4, and zstd.

Since OpenZFS 2.0, zstd is the generally recommended algorithm for most workloads, with adjustable levels from 1 to 19 (plus zstd-fast mode, comparable in speed to lz4), offering significantly better compression rates with still very acceptable CPU overhead.

zfs set compression=zstd vault/documents       # recommended by default in 2026
zfs set compression=zstd-19 vault/archive      # more compression, more CPU
zfs set compression=lz4 vault/vms              # latency-sensitive workloads
zfs get compressratio vault/documents          # verify actual compression ratio achieved

⚠️ Never set compression=off by default "to gain performance" — in practice compression almost always reduces physical I/O more than it adds CPU overhead.

Deduplication requires careful planning of RAM capacity before being enabled.

⚠️ Traditional dedup (pre-2.3): the DDT table needs to reside almost entirely in RAM/ARC for acceptable performance — historical rule of ~5GB RAM per deduplicated TB, easy to underestimate. Without enough RAM, writes and even zpool destroy can hang the system for hours. Practical recommendation: avoid traditional dedup in production, except in very specific cases with generous dedicated RAM.

✅ Fast Dedup (OpenZFS 2.3+): drastically reduces memory overhead and accelerates deletion of deduplicated blocks, making dedup more viable — still requires capacity planning before enabling.

# Simulate POTENTIAL space savings without actually enabling (does not consume space)
zdb -S vault

# Enable dedup only after validating available RAM
zfs set dedup=on vault/vms

Replication with Send/Receive

The zfs send/zfs receive pair enables efficient dataset replication, operating directly on ZFS blocks. Incremental streams transmit only the changed blocks between two snapshots, enabling fast backups and disaster recovery even on large datasets.

In practice, full replication covers from initial send to production scenarios like incremental replication, sending encrypted data, and recovery of interrupted transfers:

# Full snapshot (full send) — first replication
zfs snapshot vault/documents@full-20260623
zfs send vault/documents@full-20260623 | ssh user@remoto zfs receive backup/vault/documents

# Incremental send — only changes between two snapshots (real daily backup flow)
zfs snapshot vault/documents@inc-20260624
zfs send -i vault/documents@full-20260623 vault/documents@inc-20260624 | ssh user@remoto zfs receive backup/vault/documents

# Raw send — replicates encrypted data without ever decrypting it (untrusted offsite/cloud)
zfs send -w vault/dados-sensiveis@snap1 | ssh user@remoto zfs receive backup/dados-sensiveis

# Show progress during send (large datasets)
zfs send -v vault/documents@snap1 | pv | ssh user@remoto zfs receive backup/vault/documents

# Resumable send — resumes interrupted transfers (essential on unstable WAN links)
zfs send -t <token-of-interrupted-receive> | ssh user@remoto zfs receive -s backup/vault/documents

Integration with syncoid (part of sanoid) automates the entire cycle of snapshot creation, incremental send, and cleanup of old snapshots — today the de facto market standard tool for this flow, surpassing in adoption the older alternative zrep:

syncoid vault/documents user@remoto:backup/vault/documents

RAID-Z and Pool Expansion

RAID-Z solves the "write hole" of traditional RAID-5/6 by integrating parity directly into ZFS's transactional copy-on-write architecture. It supports up to three parity levels (Z1/Z2/Z3).

RAID-Z expansion was officially released in OpenZFS 2.3, allowing adding disks to an existing RAID-Z vdev without downtime via zpool attach:

# Add a disk to an existing RAID-Z vdev without recreating the pool
zpool attach vault raidz2-0 /dev/sde

⚠️ Expansion does not retroactively rewrite parity of old data — only data written after expansion uses the new parity ratio. Run a full scrub immediately after any expansion.

dRAID is a distributed vdev with embedded hot spares, created to accelerate resilver in large arrays. Traditional resilver with 16TB+ disks can take days leaving the pool degraded; dRAID distributes the rebuild load across all disks in the vdev:

zpool create vault draid2:4d:1s /dev/sd[a-h]
# draid2 = 2 parity, 4d = 4 data disks per group, 1s = 1 distributed spare

Native Encryption

Available since OpenZFS 0.8, native per-dataset encryption is essential today for compliance (e.g., LGPD) and requires no external layer like LUKS:

# Create encrypted dataset with password
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase vault/dados-sensiveis

# Using key file instead of password (better for automation)
zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///root/zfs.key vault/dados-sensiveis

# Load key and mount encrypted dataset after reboot
zfs load-key vault/dados-sensiveis
zfs mount vault/dados-sensiveis

# Load all keys automatically at boot
zfs load-key -a

# Change encryption password/key
zfs change-key vault/dados-sensiveis

Encryption is per dataset (not per whole pool), allowing different policies per workload, and works perfectly with the raw send shown above — replicating encrypted data without ever decrypting it on the destination.

Special vdev and Allocation Classes

Allows directing metadata and small files to a separate SSD/NVMe vdev, drastically accelerating operations with many small files (ls, incremental backups, databases with many indexes):

# Add a mirrored "special" vdev to the existing pool
zpool add vault special mirror /dev/nvme0n1 /dev/nvme1n1

# Set the file size threshold that goes to the special vdev
zfs set special_small_blocks=32K vault/documents

⚠️ A special vdev without its own redundancy makes the entire pool vulnerable if it fails — always use mirror, never single disk, for special vdevs in production.

Direct I/O

Allows reads/writes to bypass ARC directly, improving performance on NVMe workloads where duplicated ARC cache + application cache (e.g., PostgreSQL's shared_buffers) is redundant and wastes RAM:

zfs set direct=standard vault/postgres   # application decides via O_DIRECT
zfs set direct=always vault/scratch-nvme # forces Direct I/O always
zfs set direct=disabled vault/documents  # classic behavior (default)

Use Cases

File Servers and NAS

ZFS is ideal for storage servers and NAS, thanks to checksum-verified integrity, snapshots, compression, and ease of administration. Solutions like TrueNAS (unified successor of FreeNAS), OpenMediaVault, and ZFS on Linux on Ubuntu/Proxmox are widely used in production.

Databases and High I/O Workloads

The recordsize property should be adjusted to improve performance with databases, aligning the ZFS block to the application's I/O pattern (typically 8K–16K for most RDBMS, instead of the generic default of 128K). NVMe SSDs as SLOG accelerate synchronous writes; L2ARC extends read cache capacity; Direct I/O avoids duplicate caching between ARC and application buffer.

Virtualization

ZVOLs can be used as block disks for VMs (Proxmox VE, KVM, bhyve). Snapshots and clones allow near-instant provisioning from templates and consistent backups without pausing the VM for long periods. For large arrays of spinning disks used as datastore, dRAID is a relevant alternative to traditional RAID-Z.

Backups and Replication

The combination of frequent snapshots and incremental replication via send/receive offers robust local and remote backup with low overhead. For compliance (LGPD), combining native encryption with raw send keeps data end-to-end encrypted, even replicating to external destinations.

Practical Administration

Pool and Dataset Creation

# Always declare ashift explicitly at pool creation
zpool create -o ashift=12 -m /mnt/vault vault mirror /dev/sda /dev/sdb

# Dataset with recommended properties at creation
zfs create -o compression=zstd -o atime=off -o xattr=sa vault/documents

Property Management

zfs set compression=zstd vault/documents
zfs set quota=100G vault/documents
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase vault/dados-sensiveis

Snapshots and Replication

zfs snapshot vault/documents@backup-$(date +%Y%m%d)
zfs send vault/documents@backup-20260623 | ssh user@remote zfs receive backup/vault/documents
zfs send -i vault/documents@backup-20260622 vault/documents@backup-20260623 | ssh user@remote zfs receive backup/vault/documents

Maintenance and Integrity Verification — zpool scrub

scrub is the operation that effectively scans the pool, verifies all checksums, and repairs bit rot using available redundancy. Without periodic scrub, silently corrupted data may never be detected until it's too late (e.g., when another disk fails and the remaining redundancy was already corrupted).

# Start manual scrub
zpool scrub vault

# Check progress
zpool status -v vault

# Pause/resume scrub (useful in production under load)
zpool scrub -p vault     # pause
zpool scrub vault        # resume from where it stopped

# Stop completely
zpool scrub -s vault

Recommended scheduling:

# /etc/cron.d/zfs-scrub — SATA/SAS disks in production: monthly
0 3 1 * * root /sbin/zpool scrub vault

# Critical environments or enterprise SSD: biweekly
0 3 */14 * * root /sbin/zpool scrub vault

Most distros with ZFS module already include a systemd timer (zfs-scrub-weekly.timer or similar) — check with systemctl list-timers | grep zfs.

Pool Health Monitoring

zpool status -v vault                                    # overall status — first thing to check
zpool events -v                                           # recent events/errors
zpool iostat -v vault 2                                   # real-time I/O statistics
zpool list -o name,size,alloc,free,frag,cap,health         # overview of all pools
zpool history vault                                        # command history (audit)

Disk Replacement and Failure Recovery

# Replace failed disk with a new one
zpool replace vault /dev/sda /dev/sdc

# Mark disk as automatic hot spare
zpool add vault spare /dev/sdd

# Take disk offline manually (planned maintenance)
zpool offline vault /dev/sda
zpool online vault /dev/sda

# Clear error counters after resolving the physical issue
zpool clear vault

# Import pool after moving disks to another server
zpool import                      # lists importable pools
zpool import vault                # imports by name
zpool import -f vault             # force import (pool was not cleanly exported)

# Export pool properly before moving disks
zpool export vault

Performance and Tuning

• ARC: monitor with arc_summary and tune zfs_arc_max — essential on servers that also host other workloads (databases, hypervisors), where an unbounded ARC competes for memory and contributes to swap and degradation.

• L2ARC: high endurance SSDs (high TBW); recent OpenZFS versions support persistence across reboots.

• SLOG: ultra-low latency devices (NVMe SSD, Optane) for synchronous writes.

• Compression: zstd is the current default recommendation; lz4/zstd-fast for extremely CPU-latency-sensitive workloads.

• Recordsize: 8K–16K for databases and VMs; 128K or larger for large sequential files.

• Ashift: declare explicitly at pool creation (12 for 4K — most of the market; 13 for some enterprise 8K NVMe).

• Direct I/O: consider for NVMe with application-owned cache.

• Special vdev: consider for pools with a large volume of small files.

ARC Tuning in Practice

# Check current usage and detailed ARC statistics
arc_summary

# Limit ARC to a maximum (e.g., 16GB) — essential with other workloads on the same server
echo "options zfs zfs_arc_max=17179869184" >> /etc/modprobe.d/zfs.conf   # 16GB in bytes
update-initramfs -u   # Debian/Ubuntu: necessary to apply at boot

# Adjust at runtime without reboot (Linux)
echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max

# Set ARC minimum (prevents it from shrinking too much under memory pressure)
echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_min

Practical sizing rule: on dedicated storage servers, ARC can safely use up to 50% of RAM. On servers that also host applications with their own cache, limit ARC explicitly — otherwise it competes for memory with the workload, causing swap and degradation (common scenario in "slow server" diagnostics in environments with Oracle/WinThor, for example).

Quick Reference — Essential Commands

# Pool
zpool create -o ashift=12 nome mirror /dev/sda /dev/sdb
zpool status -v
zpool scrub nome
zpool replace nome /dev/old /dev/new
zpool list -o name,size,alloc,free,frag,cap,health
zpool iostat -v 2
zpool events -v
zpool import / zpool export nome

# Dataset
zfs create -o compression=zstd -o atime=off nome/dataset
zfs set quota=100G nome/dataset
zfs get all nome/dataset

# Snapshot/Clone
zfs snapshot nome/dataset@tag
zfs rollback nome/dataset@tag
zfs clone nome/dataset@tag nome/clone
zfs promote nome/clone

# Replication
zfs send -i @snap1 nome/dataset@snap2 | ssh host zfs receive destino/dataset
zfs send -w nome/dataset@snap1 | ssh host zfs receive destino/dataset   # raw send (encrypted)
syncoid nome/dataset user@remoto:destino/dataset                        # recommended automation

# Encryption
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase nome/dataset
zfs load-key -a

# Monitoring
arc_summary
zdb -S nome   # simulate dedup savings before enabling

Final Considerations

ZFS is a mature, scalable file system packed with features that ensure security and efficiency in data storage — and it continues to evolve actively: the OpenZFS 2.x series brought features like Fast Dedup, RAID-Z expansion without downtime, Direct I/O, and persistent L2ARC, which change practical design decisions that were fixed a few years ago. Its active community and comprehensive documentation facilitate adoption in environments of all sizes, from home NAS to enterprise production clusters.

For continuous deepening, the most reliable reference is always the official OpenZFS documentation, complemented by materials like the book "FreeBSD Mastery: ZFS" by Michael W. Lucas — keeping in mind that, given the pace of project evolution, it is always prudent to confirm version-specific behaviors in the official documentation before applying critical configurations in production.