Filesystem corruption on critical servers: how to diagnose before repairing

When a filesystem presents an error on a production server — especially in environments with Oracle databases, ERPs, or hospital systems —, the most common instinct is to run xfs_repair or fsck -y immediately and hope the problem goes away. This is exactly the kind of hasty decision that turns a one-off corruption into irreversible data loss.

In this article, we show the diagnostic flow we apply at Nyverra before any filesystem repair: why the order of operations matters, how to isolate whether the problem is physical disk or logical corruption, and the commands that truly separate a safe repair from a disaster.

Why not run repair right away

xfs_repair and fsck are last-resort tools — they rebuild metadata, move orphaned files to lost+found, and, in extreme cases, discard journal transactions. Running these tools without understanding the root cause of the problem has two risks:

1. Running on a disk with real physical failure. If the problem is a bad sector or a degrading controller, repair may rewrite metadata over still-recoverable data, or simply fail midway, leaving the filesystem in a worse state than before.

2. Running with the filesystem mounted or in active production. Some xfs_repair flags, like -d, exist precisely to allow repair on a filesystem that the operating system considers mounted — and using them outside a rescue/live boot scenario is an almost guaranteed recipe for worsening corruption, especially when the volume in question is the system's root partition.

Therefore, our first step is never repair. It's disk health.

Step 1 — Rule out hardware problems with SMART

Before touching any filesystem, we validate the physical health of the disk:

smartctl -s on /dev/sda
smartctl -H /dev/sda
smartctl -a /dev/sda

The attributes that matter most in this check, especially on SSDs — increasingly common in edge servers and virtualization environments —, are:

• Reallocated_Event_Count — remapped sectors. If greater than zero and growing, it's a sign of media degradation.

• Reported_Uncorrect — read errors that the disk itself could not correct.

• UDMA_CRC_Error_Count / SATA_CRC_Error_Count — usually points to a cable or connector problem, not the disk itself.

• Media_Wearout_Indicator / SSD_Life_Left — accumulated NAND wear.

• Unsafe_Shutdown_Count / Power-Off_Retract_Count — how many times the disk suffered a power cut without a clean shutdown. This number is the most direct link between hardware history and journal corruption in XFS or ext4.

If static reading is no longer sufficient, we run an active self-test:

smartctl -t short /dev/sda
# wait for the estimated time before checking
smartctl -l selftest /dev/sda

The result we look for is Completed without error. Any variation of that — read failure, timeout, abort — already indicates the problem is in the disk, and in that scenario the path is hardware replacement before any logical repair attempt, not the other way around.

Step 2 — Cross-reference with unclean shutdown history

In virtualized or test environments, it's common to find a high Unsafe_Shutdown_Count even on healthy disks — the value accumulates with every VM reset, power outage without UPS, or kill of a hypervisor process without going through a clean ACPI shutdown.

This is the classic pattern of XFS corruption due to incomplete journal: the disk is physically intact, but the filesystem ended up in an inconsistent state because the last write operation was not correctly committed. It's worth supplementing with:

dmesg | grep -i xfs
dmesg | grep -i -E "ata|error"

If messages of metadata corruption or I/O error appear near the time of the incident, and SMART is clean, the cause is practically confirmed: unclean shutdown, not hardware.

Step 3 — Filesystem diagnosis, always in dry-run mode

Only after ruling out hardware do we get into the filesystem itself — and always starting with the mode that changes nothing:

xfs_repair -n /dev/mapper/vg_data-lv_data

The -n reports everything that xfs_repair would find and fix, without writing a single change to disk. It is this output that decides the next step: if the problem is superficial (secondary metadata, free block counters), standard repair resolves it without drama. If the log is really corrupted, the behavior changes.

Step 4 — The repair, with the filesystem unmounted

Ideal environment: filesystem unmounted, or system in rescue/live boot when it comes to the root partition.

umount /dev/mapper/vg_data-lv_data
xfs_repair /dev/mapper/vg_data-lv_data

If xfs_repair refuses to run because of a dirty log, the first attempt is to let the kernel replay the journal normally:

mount -t xfs /dev/mapper/vg_data-lv_data /mnt/tmp
umount /mnt/tmp
xfs_repair /dev/mapper/vg_data-lv_data

Only when that is not possible — the log is corrupted to the point of not even allowing mount — do we consider zeroing the log:

xfs_repair -L /dev/mapper/vg_data-lv_data

This flag discards any uncommitted transaction. It is an operation with potential for loss of recent data and should be treated as a last resort, never as a first attempt — and ideally preceded by an LVM snapshot, when the infrastructure allows:

lvcreate -s -L 10G -n snap_before_repair /dev/vg_data/lv_data

And on ext2/ext3/ext4 filesystems?

The logic is the same, with fsck or e2fsck:

fsck -n /dev/sdb1                 # diagnostic, without changing anything
umount /dev/sdb1
fsck -f -y /dev/sdb1              # forced repair, automatically accepting corrections

The structural difference is that XFS does not have a traditional "fsck mode" — xfs_repair is the repair tool itself, and journaling recovery normally happens automatically at mount. In ext4, however, fsck is interactive by nature, and -y serves precisely to automate responses in unattended executions or boot scripts.

The checklist we apply before any repair

1. Unmount the filesystem, or operate via rescue/live boot when it is the root partition

2. LVM snapshot, whenever the infrastructure allows

3. Complete SMART check of the physical disk first

4. Active self-test if history or attributes raise doubt

5. Diagnosis in dry-run mode (-n) of the filesystem before actual repair

6. Standard repair, escalating to -L (XFS) only as last resort

7. Check of lost+found after repair

Why this matters in critical environments

In healthcare and industry infrastructure — the sectors in which Nyverra operates most intensely —, a corrupted filesystem is not just a technical inconvenience. It can mean unavailability of electronic health record systems, interruption of a production line monitored by SCADA, or loss of regulatory data. The difference between an incident resolved in minutes and a disaster recovery taking several hours almost always lies in diagnostic discipline: understanding the cause before applying the fix.

Filesystem repair is not a "try and see what happens" operation — it is a sequence of decisions that depends on evidence, not on haste.

Nyverra specializes in critical IT infrastructure, virtualization, and business continuity for B2B clients in the healthcare and industry sectors. Learn more at nyverra.com.br.