How SMART tells you when to replace a drive

SMART does not give a drive an exact date of death. It shows traces of failures that have already happened, wear, and several states in which the drive itself admits that it can no longer store data reliably. Immediate replacement is therefore warranted by an overall FAILED status, active unreadable areas, a failed self-test, or a critical NVMe state, not by every yellow indicator.

The most dangerous administrator error can look perfectly rational: notice a warning, run a long test or a full overwrite, and only then deal with the backup. If the data matters, reverse that order. Stop unnecessary writes and save whatever is accessible before refining the diagnosis. A green SMART status never replaces a backup, and RAID does not turn a sick drive into a healthy one.

Immediate replacement starts with four signals

Remove a drive from its production workload immediately if it has reported failure, can no longer read data consistently, failed an internal read test, or entered a critical NVMe state. "Immediately" means that you do not wait several weeks for a maintenance window, start a stress test, or watch to see whether the counter rises again. Protect the data and prepare a replacement.

For HDDs and SATA SSDs, I use four groups of red signals:

• SMART overall-health self-assessment test result: FAILED or SMART Health Status: BAD;
• a nonzero and rising Current_Pending_Sector or Offline_Uncorrectable alongside read errors;
• a Completed: read failure entry in the self-test log;
• repeated I/O errors, command hangs, or drive disappearances that coincide with errors reported by the media itself.

For NVMe, the main red flag is called critical_warning. The degraded reliability and read-only bits state directly that the subsystem has lost reliability or placed all media in read-only mode. When available_spare drops below the vendor-set threshold, the drive also sets a critical bit. Remove such a device from service in a server even if applications can still read most of the data.

There is an important qualification: one reallocated sector that remains stable, with no other errors, does not prove that a particular HDD will fail soon. You can sometimes monitor it in noncritical secondary storage at home. In a production server with a clear maintenance window, I still plan a replacement because repeated diagnosis and array degradation often cost more than a drive. The active error, its trend, and the drive's role determine urgency.

Read RAW, VALUE, and history separately

Reading SMART correctly starts by separating the raw counter, the normalized assessment, and the vendor threshold. These columns answer different questions. Mixing them up causes most false alarms.

In a typical ATA report, VALUE starts around 100 or 200 and usually falls as the condition worsens. WORST retains the lowest normalized value, while THRESH sets the boundary below which the attribute has failed. RAW_VALUE shows an internal counter, but the vendor chooses its format. A value of 100 in VALUE does not mean 100 percent health, and a large RAW_VALUE does not always represent hundreds of physical errors.

This is particularly obvious with Raw_Read_Error_Rate and Seek_Error_Rate. Some models encode several quantities in the raw field or count a huge number of successful operations together with errors. Seagate's own guidance explicitly warns that individual attribute values and thresholds are proprietary and that third-party programs can interpret them inconsistently. Do not apply a threshold from a random table to a different model.

People also misread the WHEN_FAILED column. A blank field does not promise that the drive is healthy. It only says that the normalized attribute has not crossed the vendor's threshold. SMART can display PASSED while the log already contains a read error or several pending sectors. The smartmontools code deliberately distinguishes an overall status failure, an attribute below threshold, errors in the error log, and errors in the self-test log.

History matters more than a single snapshot. Save the model, serial number, power-on hours, all attributes, and the test log, then compare that drive with itself. Compare RAW values across drive families only after checking vendor documentation. If Current_Pending_Sector was 0 yesterday, becomes 1 today, and rises to 8 after ordinary reading, the surface is actively deteriorating even if normalized VALUE still looks reassuring.

Pending and uncorrectable sectors put an HDD at risk

Attributes 5, 187, 197, and 198 are the most useful on a magnetic drive, but each describes a different stage of failure. They are not interchangeable.

Reallocated_Sector_Ct (5) shows how many bad physical sectors the drive has already replaced with spares. Data at the logical address may be readable again, so this is a historical trace rather than necessarily a current error. One stable reallocated sector calls for monitoring. Rapid growth, new read errors, or shrinking normalized reserve calls for replacement.

Current_Pending_Sector (197) is more dangerous: the drive could not read a sector confidently and is waiting to decide what to do with it. A later successful write may confirm that the sector works or make the firmware reallocate it. Until then, data at that LBA may be unavailable. A nonzero 197 on a drive holding the only copy of data means copying everything readable first, not trying to "clear pending."

Offline_Uncorrectable (198) counts areas that the drive could not correct during background scanning or a self-test. If 198 is nonzero and the log contains read failure, the media has already shown a reproducible reading problem. I do not leave that HDD in a production array after its formal status has been restored.

Reported_Uncorrect (187) records errors that the drive could not correct and reported to the host. Command_Timeout (188) marks commands aborted on timeout, but you must compare it with power, controller, and model information. Backblaze selected attributes 5, 187, 188, 197, and 198 to investigate potential failures across its large HDD fleet. That is a useful empirical filter, not a universal guarantee: its policy covers arrays where a drive can be replaced without losing the only copy.

The Google study "Failure Trends in a Large Disk Drive Population" reaches a less comfortable conclusion. Reallocations, scan errors, and pending sectors correlated well with failures, yet many failed drives showed none of the strong signals. SMART confirms risk well after counters light up, but zeros do a poor job of proving that there is no risk.

NVMe has its own end-of-life signals

Do not look for familiar numbered HDD attributes on NVMe. The standard defines a common SMART / Health Information page with clearer states: critical_warning, available_spare, available_spare_threshold, percentage_used, media_errors, and an error log.

The NVM Express 2.0 specification defines separate critical_warning bits. Bit 0 means that available spare has fallen below its threshold; bit 1 reports a temperature outside a specified boundary; bit 2 marks degraded reliability caused by serious media or internal errors; bit 3 says that all media has entered read-only mode. Two more states cover devices with backup power and persistent memory.

The bits do not have equal urgency. Read-only and degraded reliability require immediate removal from writes and replacement. Spare below threshold means that reserve has reached a level the vendor itself calls critical, so waiting for another warning makes no sense. A temperature bit first calls for fixing cooling and reducing load, but a repeated warning after temperatures normalize points to a device or sensor problem.

percentage_used is firmware's estimate of consumed endurance based on expected write life. A value of 100 does not mean that the drive will switch off that minute, and some devices allow values above 100. It means that rated write endurance has been consumed. This is a planned replacement boundary in production, while simultaneous media_errors, a critical warning, or application errors make the replacement urgent.

media_errors counts unrecovered integrity errors, including uncorrectable ECC, a CRC failure within the path defined by the standard, or an LBA tag mismatch. You cannot assess one old count after a known incident without the error log and trend. New growth during ordinary load, especially with a critical warning, cannot be dismissed as age. By contrast, unsafe_shutdowns records a loss of power without a normal shutdown notification. It explains operating conditions but does not prove NAND wear by itself.

Cable faults and heat are not media damage

Interface, power, and temperature counters call for repair of the path, not automatic disposal of the drive. They help you avoid replacing a healthy drive while leaving the actual defect in the server.

For SATA, UDMA_CRC_Error_Count (199) usually rises when data is corrupted in transit between the drive and controller. Common causes are in the cable, connector, drive bay, power, or port. An old nonzero value does not reset after a cable replacement. Record it, reseat or replace the path, and watch whether RAW continues to rise. If the count stops, the media may be fine. If 197 and 198 rise at the same time, replacing the cable alone does not close the diagnosis.

High temperature calls for action, but there is no universal number for every model. For NVMe, compare the current temperature with the warning and critical thresholds in Identify Controller. For SATA, consult the manual and specification for that model and review the history in the same bay. A sudden rise on one drive while its neighbors remain normal is more suspicious than the whole row warming after a fan failure.

Power_On_Hours, Power_Cycle_Count, Start_Stop_Count, and Load_Cycle_Count describe age and operating patterns. A large number of hours helps plan a fleet refresh but does not create an instant verdict. The same applies to Wear_Leveling_Count and proprietary life-remaining indicators on SATA SSDs: check the model documentation for the meaning and direction of the scale.

Do not ignore an interface error counter just because it is "not about the disk." A bad connector can cause hangs, eject a disk from RAID, and damage an unfinished write. The remedy is simply different: save the data, repair the cable, power, or backplane, then check the trend again. Replacing a drive without repairing the path often produces the same symptoms on the new device.

Status and trend make the decision together

A single SMART snapshot warrants action by itself only when it shows an explicit failure. In every other case, base the decision on current impact to data, counter growth, and the drive's role.

• Overall FAILED or BAD: firmware has crossed its own failure threshold. Remove the load, copy what is available, and replace the drive immediately.
• NVMe critical warning with read-only or reliability degraded: the controller has acknowledged lost reliability. Do not return the device to writes, and replace it.
• Attribute 197 or 198 is rising and a self-test reports read failure: an active unreadable area exists. Copy before tests, then replace the drive.
• Attribute 5 is nonzero and stable with no other errors: the drive has already reallocated an area. Increase monitoring and plan replacement for a critical role.
• Attribute 199 is rising without media errors: a SATA path fault is more likely than a surface defect. Repair the cable, port, bay, or power and measure again.

The guidance is intentionally conservative in some cases. Stop troubling a drive with the only copy of an accounting database sooner than a drive with another replica and a verified backup. After one member of a RAID has failed, the remaining drives face heavy reading during rebuild, so several pending sectors on one of them are more dangerous than on a separate archive device.

I do not use a rule such as "replace after ten bad sectors." It has no common technical basis because sector size, firmware reserve, model, rate of growth, and error behavior differ. Zero does not guarantee survival, and a chosen round number does not make risk manageable. The fact that data is unavailable matters for active pending and uncorrectable sectors; trend and context matter more for reallocated sectors.

Consider a mirror made from two HDDs. The array already marks the first drive as failed, and the second has developed two pending sectors. An automatic rebuild onto a new drive will start sequentially reading the second disk and will almost certainly reach the suspect area. If the only backup has not been verified, pressing rebuild without thought turns a controlled degradation into a race between reading and another error. Confirm the backup, preserve irreplaceable data, and identify which source can still be read.

Copy priority also depends on the error. If the directory opens but large archives return I/O errors, take small unique documents and application metadata first, followed by bulky data that can be recreated. Repeatedly reading one bad file delays everything else. Tools such as ddrescue keep a map of read and skipped areas, so they can collect the easy part of an image first and return to difficult blocks later. That is not drive repair; it manages repeated access.

Save data before checking the surface

When a red signal appears, the right order reduces reads from sick media and preserves the chance of recovering the most important files. Diagnosis must not compete with data evacuation.

1. Stop applications, indexing, defragmentation, scrubs, and jobs that write to the disk. If the failure affects an array, follow the storage procedure rather than pulling a device at random.
2. Capture the full SMART report, kernel log, model, serial number, slot, and event time. Do not spend hours on this if the drive is already disappearing.
3. Copy the most important and unique data to healthy storage. When reads are unstable, use an imaging tool that can skip bad areas and return to them instead of ordinary copying with endless retries.
4. Verify the resulting copy: open critical files, compare known hashes, and run the application or database check against the copy.
5. Replace the device, restore redundancy, and only then investigate the removed disk if analysis is still useful.

The popular advice to write zeros to a pending sector really can make firmware either accept or reallocate it. That is exactly why it is dangerous before copying: the write destroys the previous contents of the LBA and changes the evidence. After successful recovery, such an operation is suitable for a lab check, but it does not restore trust in a drive that produced an uncorrectable error under production load.

A long SMART self-test mostly reads the surface inside the drive and is useful on stable media. On a drive that clicks, disconnects, or rapidly adds pending sectors, the test adds work and delays copying. If the data is irreplaceable and ordinary reading makes the condition worse, stop experimenting and send the media to data-recovery specialists.

A full report beats a colored indicator

Initial diagnosis needs a full text report, not only a green checkmark from a graphical utility. smartctl shows overall status, attributes, error logs, and self-test logs; nvme-cli can show the same health page for NVMe.

sudo smartctl -x /dev/sdX
sudo smartctl -l selftest /dev/sdX
sudo smartctl -x -j /dev/sdX > smart-sdX.json

sudo nvme smart-log /dev/nvme0
sudo smartctl -x -j /dev/nvme0 > smart-nvme0.json

A useful ATA baseline captures SMART overall-health, the attribute table, SMART Error Log, and SMART Self-test log. For NVMe, retain at least critical_warning, available_spare, available_spare_threshold, percentage_used, media_errors, num_err_log_entries, temperature, and time spent in dangerous temperature states. JSON works well for monitoring because you do not have to parse columns aligned with spaces.

You can run a short self-test after backing up if the drive remains consistently accessible. smartctl -t short /dev/sdX usually only schedules the test; read the result later with smartctl -l selftest /dev/sdX. Use -t long for a full read, and the utility reports the expected duration. A clean short test does not cover the whole surface.

The smartctl exit status is a bitmask, not a conventional zero or one. In the smartmontools documentation, bit 3 means that overall SMART status failed, bit 4 reports a current prefail attribute below threshold, bit 6 indicates entries in the error log, and bit 7 indicates errors in the self-test log. A script that checks only exit_code == 1 will miss serious conditions. Save the JSON and exit status first, then parse the relevant bits for the device type.

A RAID controller or USB bridge can hide SMART or require a device-type parameter. If smartctl cannot see a drive behind an HBA, that does not prove health. Request the data through the controller's supported interface or BMC, map the logical drive to its physical slot, and do not replace a device based only on a /dev/sdX name that may have changed after a reboot.

Read the ATA error log together with power-on time and command type. An old entry may predate a cable replacement and remain forever, while a fresh UNC error at the same LBA confirms a reading problem. A self-test marked Interrupted (host reset) is not the same as Completed: read failure: a reboot, sleep mode, or controller command may have interrupted the first. Repeat it only after finding the reset cause and protecting the data.

For NVMe, num_err_log_entries does not mean the number of lost files either. It counts entries in the controller error log, which can contain different command statuses. The decision needs the entries themselves, media_errors, critical warning, and operating system events. If a faulty administrative command from a utility increases the counter, replacement is unnecessary; if new entries coincide with an I/O error and a media error, do not postpone it.

Monitoring should catch transitions, not pretty numbers

A useful alert identifies new risk and its speed rather than emailing the same old counters every day. Keep a baseline and the latest value for every serial number. Mark the first pending or uncorrectable sector, media error, failed self-test, and any critical warning separately.

A sensible HDD escalation policy looks like this: overall FAILED, a new self-test read failure, and growth in 197 or 198 create an emergency event; a new 5 creates a warning and an investigation ticket; growth in 199 sends a task to inspect the path. The planned replacement threshold for 5 depends on the role, but repeated growth after the first event is no longer an incidental old mark.

For NVMe, the reliability degraded, read-only, and spare below threshold bits create emergency events. Alert on percentage_used in advance according to internal purchasing policy and supply lead times, rather than for the first time at 100 percent. A temperature event should include the current value, controller thresholds, duration of overheating, and the state of neighboring devices. The on-call engineer can then distinguish a failed fan from one hot drive.

Test the monitoring itself on a healthy test device. I have seen configurations collect smartctl for months while losing the exit status, ignoring the self-test log, or changing a field name after an update. A blank chart in such a system means collection is broken, not that errors are absent.

SMART does not replace operating system monitoring either. Block-layer timeouts, controller resets, rising latency, a read-only filesystem, and RAID degradation can appear before an understandable attribute. Collect application events and media state on one timeline. You can then see what happened first: overheating, an interface error, media damage, or power loss.

Fix the cause after replacing the drive

A new drive fixes a media problem, but it does not repair a bad backplane, rack overheating, or the lack of a verified copy. After restoring redundancy, compare the SMART data from neighboring devices, controller logs, and temperature in the same slot. If CRC errors and resets move to the replacement drive, the path remains faulty.

An organization should record more than "drive replaced." Record the reason: overall FAILED, pending growth, failed self-test, NVMe critical warning, interface error, or endurance-based preventive work. This classification shows which failures SMART actually warned about and how many replacements resulted from cables or cooling. It also helps set spare inventory and a realistic service window without an invented universal threshold.

For GSE.kz servers, connect storage diagnosis with 24/7 technical support and the details of the specific configuration, including its controller, drive bay, and rack conditions. A manufacturer and system integrator can assess the entire path and replacement compatibility, while one SMART screenshot describes only part of the system.

Do not assign a removed drive as a "second backup" after it has reported an uncorrectable error or failed a self-test. Sanitize its data under the organization's policy, process a warranty return, or keep it only for noncritical lab diagnosis. The operating boundary is simple: explicit failure or active loss of readability requires replacement now; a stable historical counter requires monitoring; an interface fault requires interface repair. All three cases require an existing, verified backup.