How to run a server stress test in 24 hours

A 24-hour server stress test does not prove that a machine will run for five years. It solves a narrower and more useful problem: it forces an early manufacturing defect to appear before the server receives data and users. In 24 hours, you can test compute components, memory, storage, cooling, and power under a reproducible load, then accept the server using criteria written in advance.

The main acceptance mistake is not choosing a "weak" utility. Engineers start a workload, see 100% CPU use, and declare the server healthy several hours later. Utilization says nothing about machine checks, ECC corrections, write integrity, throttling, or the loss of one power feed. A test has value only when it includes a baseline snapshot, telemetry over time, and a rule that sends each specific deviation to a repeat test or component replacement.

How to fit a complete test into 24 hours

The goal is not to hold one workload for as long as possible. Use the day to create different operating modes and keep enough time to repeat a suspicious episode. My working schedule looks like this:

1. Hours 0-1: verify the configuration and firmware, then capture baseline BMC logs, SMART, EDAC, temperatures, fan speeds, and power supply readings.
2. Hours 1-5: run compute loads with short algorithm changes, followed by a combined CPU and memory load.
3. Hours 5-11: test all available RAM while watching ECC and the system log.
4. Hours 11-18: test each drive, first with reads and then with a verification write pattern on drives that may be overwritten.
5. Hours 18-22: load CPU, RAM, and drives together while monitoring temperatures, frequencies, fans, and power.
6. Hours 22-24: perform a controlled power redundancy check, cool the server, repeat a suspicious test, and capture the final logs.

This schedule does not require every server to perform the same number of operations. A dual-socket machine with a large amount of RAM and dozens of drives may not complete a full pass over every memory cell and disk sector. In that case, integrity checks, controller errors, and the combined peak load take priority. Record the amount of data actually tested. The statement "memory tested" without a number of bytes hides a gap in the method.

Set four constraints before starting. First, decide whether the storage may be erased. Second, confirm whether the site permits one redundant power supply to be disconnected. Third, record the inlet air temperature. Fourth, identify the BIOS, BMC, controller firmware, and drive firmware versions approved for delivery. If any answer is unknown, record the exception in the test plan. Do not quietly replace a destructive disk test with a read-only pass and present the two modes as equivalent.

Test the server in its intended configuration: the same memory modules, HBA or RAID controller, drives, network cards, power supplies, and BIOS profile. Moving a DIMM after the test invalidates the conclusion about that specific memory channel. A firmware update after the test requires at least a short rerun because it can change power control, fan behavior, memory training, and error handling.

Do not spend the whole window installing tools. Prepare the boot image, packages, job files, and metric collection script in advance on a machine with the same architecture. Confirm that the image can see the BMC, RAID controller, NVMe drives, and EDAC, and that the system clock synchronizes without the production network. An isolated environment may have no repository, and four acceptance hours can disappear while someone searches for dependencies. Tool versions belong in the report too: different smartctl and stress-ng releases recognize new devices and sensors differently.

A baseline separates a new defect from an old record

Save the server state before applying load so that you can calculate the increase in every counter afterward. An absolute error count without a baseline is often misleading. The SEL may contain an event created during assembly while a fan was disconnected, and an NVMe Error Information counter may contain records unrelated to media damage. The decision should depend on a new record created during your test, its type, and its relationship to the workload.

Record serial numbers and physical component locations. A DIMM needs a socket, channel, and slot mapping; a drive needs a bay and controller identifier; a power supply needs a PSU1 or PSU2 position. One lshw snapshot does not replace this map. The operating system may show a device without providing the physical address a technician needs to find the failed part.

On Linux, capture a minimal baseline with these commands:

ipmitool sensor list
ipmitool sel elist
smartctl -x /dev/sda
smartctl -x /dev/nvme0
journalctl -k -b

ipmitool sel elist is more useful than the short SEL list because the extended output cross-references an event with its Sensor Data Record and names the sensor. The IPMItool manual describes elist in exactly this way. Save the raw output, the BMC time, and the system time. If the clocks disagree, a sequence of "overtemperature first, reset second" can easily be read in reverse.

For ECC, capture EDAC counters before and after the test or collect events with rasdaemon. The Linux kernel documentation separates corrected, uncorrected, deferred, and fatal errors. These are not four names for the same event. A corrected error means ECC repaired the data, while an uncorrected error means correction failed even if the system continued running. A deferred error may surface later when software touches poisoned memory. A single "hardware errors" total is therefore useless in a report; keep the categories separate.

Do not clear SEL and SMART before saving the baseline snapshot. Once it is saved, you may clear SEL if your procedure calls for a clean window and the customer's policy permits it. The report must then contain both files: the log before clearing and the test log. SMART counters are not normally cleared, so calculate their difference.

Perform one cold boot instead of relying only on a software restart. When all power is removed, the BMC, controllers, and drives follow a different initialization path, memory trains again, and redundant power supplies negotiate load sharing. This is where a hanging POST, missing channel, slow drive start, or cache battery and supercapacitor error may appear. If the procedure prohibits power removal, record that cold boot was not tested. A successful warm restart does not cover that risk.

Record inlet air temperature, not only the CPU sensor. You cannot fairly compare a server tested beside an open door at 18 °C with one in a dense rack at 27 °C. Take upper limits from the documentation for the specific platform and processor. A universal rule such as "anything below 80 °C is fine" is poor practice because sensors differ in location, permitted limit, and throttling behavior.

Judge the processor by errors and sustained frequency

A healthy CPU should finish varied compute workloads without MCEs, hangs, or verification errors, while maintaining the expected frequency behavior after reaching thermal equilibrium. One algorithm uses execution units differently from another. Ten hours of identical integer operations reveals less than several modes that exercise matrices, integers, floating point, and cache.

For the first phase, I use stress-ng. Its manual states that verify checks results for stressors that support it, while thermalstat collects readings from available Linux thermal zones. Read the qualification carefully: verify does not work for every stressor, and verification itself reduces the operation count. A successful exit code does not turn every workload into a functional test.

A simple two-hour run on all logical CPUs can begin with:

stress-ng -c 0 -t 2h

At the same time, collect per-socket frequencies, package power, temperatures, fan speed, and kernel events every 10-30 seconds. Do not settle for an average. A poorly seated heatsink or bad thermal compound often appears as one hot socket, earlier frequency reduction, and faster fans in that zone. An average across two sockets hides the imbalance.

Machine Check Architecture helps interpret these episodes. Intel documentation lists system bus, ECC, parity, cache, and TLB errors among the hardware events it records. An MCE does not automatically condemn the processor because the source may be memory, a bus, or power. However, a new uncorrected or fatal machine check during acceptance always moves the server to "not accepted" until the cause is isolated. An unexplained restart also counts as a failure even if the second run passes.

Frequency needs context. Compare it with the expected operating point for the number of active cores, configured power limit, BIOS profile, and inlet air temperature, not the advertised single-core maximum. A brief dip when the workload changes phase is normal. A sustained sawtooth pattern combined with a thermal trip, BMC event, or sudden fan increase points to cooling. Consistently low power without overheating more often leads to a BIOS profile, power limit, or firmware setting.

Repeatability separates a defect from random noise. If an error appears 40 minutes into a matrix workload on one socket, cool the server, rerun the same mode, and record the time to the event. Then, if the design and warranty procedure permit it, move only one suspected component. Do not move the CPU, DIMM, and power supply together because a successful rerun will localize nothing.

Stop the CPU phase for an uncorrected or fatal MCE, a computation error reported by the utility, a hang, an unexpected restart, a temperature beyond the platform specification, or repeatable throttling absent from a reference configuration under the same conditions. One corrected record requires investigation and a rerun, not an automatic pass.

Memory must provide more evidence than zero utility errors

RAM testing writes and reads patterns across as much available memory as possible while watching ECC at the memory controller. Zero errors in a user-space program does not prove clean memory because ECC may correct a bit before the process reads it. The opposite misdiagnosis also occurs when the OOM killer ends the test or a hypervisor reclaims memory and a technician records a DIMM failure.

An installed operating system cannot allocate every physical byte. The kernel, drivers, and test program retain some RAM. For initial acceptance, combine a long stress-ng virtual memory pass with a bootable memory test if the plan allows a restart. The bootable test sees more address space, while Linux exposes EDAC, machine checks, and real firmware behavior under combined load. The two methods answer different questions.

Before the phase, record installed and usable memory, channel layout, frequency, rank count, and whether ECC is active. Check the memory training log after a cold start. A server may boot at a reduced frequency or with a channel disabled and still report "a lot of memory." Comparison with the ordered configuration catches incorrect installation before the stress run.

Watch three independent signals during the pass:

• the program reports a mismatch or verification error with an address;
• EDAC or the BMC increments a corrected, uncorrected, deferred, or fatal counter;
• the kernel reports an MCE, page offlining, a channel fault, or a DIMM fault.

The Linux RAS documentation describes corrected errors carefully: the system keeps running because data has not yet been damaged, but preventive replacement of a module producing such events can reduce the risk of a future uncorrected error. That does not mean every historical CE requires discarding a DIMM. For a new server, a practical acceptance rule is stricter: zero new CEs during the test window. A repeatable new CE tied to one DIMM or channel justifies module replacement or investigation of the socket and board even when the application stays up.

Uncorrected, deferred, and fatal errors stop acceptance. First save the logs, address, syndrome, DIMM label, and load phase. Then perform another cold start and rerun the test in a controlled configuration. If the event follows a DIMM after an approved swap, replace the module. If it stays with a channel or socket, investigate the connector, the processor's memory controller, or the system board. This isolation prevents repeat visits: replacing the first available memory stick does not repair a bent socket contact.

Use memory speed as diagnostic context, not a universal pass threshold. NUMA placement, channel count, ranks, power mode, and test algorithm all change bandwidth. It is more useful to compare symmetric nodes within the server and identical servers of the same specification. A large and repeatable deviation on one socket calls for a DIMM layout and NUMA check, but the platform owner must set the permitted percentage before the test.

Accept a drive after checking data, not a speed graph

A drive can deliver the expected IOPS while returning corrupted blocks. An acceptance test must therefore separate performance, data integrity, and media health. A latency graph answers "how does the device service requests," verification answers "did the same bytes come back," and SMART plus controller logs explain hardware events. None of these sources replaces the others.

Start by capturing smartctl -x for every physical drive. RAID requires the syntax and utility for its specific controller because /dev/sda may be a virtual volume. For NVMe, save media and data integrity errors, available spare, critical warning, and unsafe shutdowns. For SAS and SATA, save the error log, self-test log, and reallocated, pending, and uncorrectable sector values if the device publishes them. Attribute names and meanings differ, so one threshold table cannot be applied to every protocol.

The smartctl manual makes an important point: an extended SMART test may take tens of minutes or several hours, and you must read the result in the self-test log. Starting the command is not the result:

smartctl -t long /dev/sda
smartctl -l selftest /dev/sda

On an empty drive or a dedicated test partition, fio can write a pattern and verify it during reading. The fio documentation treats verification as a separate mode and warns that a read workload with verify expects a file written earlier. I use an explicit job file so the parameters remain in the report:

[global]
ioengine=libaio
direct=1
filename=/dev/nvme0n1
verify=crc32c
verify_fatal=1
group_reporting=1

[write_verify]
rw=write
bs=1M
size=100%
do_verify=1

This example destroys the contents of the named device. Before running it, a second engineer must match filename with the serial number and drive bay instead of trusting an unstable name such as /dev/nvme0n1. On a server that contains data, point the job to a prepared test file or a limited free partition, but accurately record the portion of the surface tested. A file test says nothing about blocks outside that file.

After the sequential pass, add a mixed random workload with a queue depth close to the expected production workload. Collect more than average IOPS. You need bandwidth, errors, average and tail latency, queue depth, drive temperature, and controller events. The project must set its own p99 limit because a database array and an archival HDD have different requirements. Outliers among identical drives also matter during acceptance. Repeated timeouts, controller resets, link drops, or a sharp latency increase on one drive are not an "SSD characteristic."

Controller cache changes the picture. A short write can fit entirely in protected cache and report a speed the array cannot sustain. Write more data than the cache holds and wait for steady state. Confirm that write-back mode has a battery or supercapacitor protecting it when the configuration calls for one, and check the controller log for failed protection. Do not change write-through to write-back just to improve the result. Acceptance must measure the approved safe setting.

Verification mismatch, new media or data integrity errors, uncorrectable sectors, a drive dropping from the array, or a failed extended self-test are unconditional failures. An increase in reallocated or pending sectors on a new SATA drive also calls for replacement. One NVMe error log entry without decoding is not enough because some records report an unsupported command. Inspect the status field, before and after counters, kernel log, and event timing.

Test power under a combined load

Power defects tend to appear when processors, memory, and drives demand power together and the fans are already running fast, not during an isolated CPU test. The final four load hours should therefore overlap subsystems. Run computation, memory allocation and verification, writes to approved test areas, and network traffic if the adapters are part of acceptance.

Use the BMC to collect input power, output power, current or status for each PSU, voltage, inlet and outlet temperature, fan RPM, redundancy state, and SEL. Available sensors vary by platform. Do not turn a missing reading into zero; mark it as "sensor not provided." A zero-watt reading from an operating supply may indicate a read error, BMC firmware issue, or a supply that is not actually sharing the load.

Watch the shape of each transition. Power rises when disk writing starts, fans react after a delay, and temperature reaches a plateau. If CPU frequency drops at the same time as a Power Unit redundancy lost or voltage threshold event, investigate power before cooling. If inlet temperature rises across the whole rack, hot-air recirculation may be the cause rather than the server.

A redundancy test means physically disconnecting one input only on a server with independent PSUs, independent feeds, and confirmed capacity for either supply to carry the load. A site specialist must perform it using an approved procedure. Do not pull the cable from a single source, work inside a supply, or simulate a failure in a loaded production rack. A one-day acceptance test does not justify risk to people or nearby equipment.

For an approved test, first confirm that the load is shared and each feed can carry the current peak. Disconnect one feed, verify that the server does not restart or log hardware errors, wait for the correct redundancy-loss event, restore power, and repeat on the second feed. The server must return to redundancy restored. An incorrect SEL event is also an acceptance defect because operations will not know that the server is running on one supply.

Power fluctuation alone does not justify replacement. A platform voltage threshold violation, supply failure, load loss during transfer, restart, smell or sound of an electrical defect, repeatable power fault, or inability to restore redundancy does. If you see smoke, sparks, abnormal heating, or detect an electrical smell, remove load immediately under the site's emergency procedure. Do not rerun the event just to obtain a cleaner log.

Write replacement criteria before starting

If criteria appear only after an error, a team will almost always bend them to meet the desired delivery date. Agree on a decision matrix for every subsystem before the test:

This matrix deliberately contains no rule that "any temperature above X means a defect." X belongs to the specification for that sensor and platform. There is no universal IOPS tolerance either. A median and range from reference servers help with a uniform batch, but the reference machine must pass the functional criteria too. A fast server that corrupts data is still faulty.

Distinguish component replacement from rejecting the whole server. An error that clearly follows one drive or DIMM usually leads to replacement of that part and a repeat of affected phases. An event tied to a memory channel, repeated resets on several drives behind one controller, or a simultaneous drop on two PSUs points to a shared component. Replacing endpoint devices would only hide the cause.

"Accepted with limitations" is appropriate for a documented method limitation, such as a policy that prohibited destructive writing to a prepared array. It is not appropriate for an unexplained hardware error. Write "100% of LBAs read; write verification not performed" instead of "drives passed stress testing." The customer can then decide whether to accept the remaining risk.

GSE controls the equipment lifecycle for S200 Series deliveries from manufacturing through support, so the decision matrix can be agreed alongside the configuration and acceptance protocol. The same method works for any other platform: criteria must cite the specification of the delivered component rather than an engineer's habit.

The report must reproduce every conclusion

A good report lets another engineer repeat the exact episode that produced a deviation instead of repeating the entire day. A screenshot of a green window is not enough. The report needs hardware identifiers, conditions, exact load parameters, time series, and raw logs.

In the header, record the server model and serial number, slot-by-slot configuration, BIOS, BMC, and firmware versions, performance profile, operating system and kernel, date, rack location, and inlet air temperature. List untested functions separately, such as a second network port, GPU, particular HBA, or fan hot-swap. A visible coverage gap is better than a hidden assumption.

For each phase, record start and end time, command or job file, data volume processed, completion status, and a reference within the report package to the raw log. Collect metrics against a common clock. CSV or JSON is more useful than a picture because it supports maximum searches, time plots, and comparisons between runs. Keep a BMC screenshot as supplementary evidence when it confirms the physical location of a sensor.

The telemetry collector can fail too, so check series completeness during the run. A gap covering the last 20 minutes before a restart often removes the only useful interval. Write every sample immediately to another device or management machine instead of holding it in test-server memory. Record a command exit code and a connection interruption separately: a missing row may mean a stopped sensor, hung operating system, or lost management link, and each cause requires a different response.

The summary line for each phase must contain a measurement and its criterion. "Temperatures normal" proves nothing. An entry such as "CPU1 max 76 °C at inlet max 25 °C, platform threshold 85 °C, thermal events 0" can be checked. These numbers demonstrate the report format, not universal limits for your server. Insert actual measurements and the model's official limit.

Attach baseline and final SEL, SMART, and EDAC output even when the difference is zero. For counters, state the formula delta = after - before. If SEL was cleared, attach the pre-clear log and command time. Map each drive name, WWN or serial number to its physical slot. Map each EDAC memory label to the slot marking in the service manual.

Do not edit a raw log manually. If it contains a password, management address, or other sensitive information, create a redacted copy for delivery, retain the original in protected storage, and describe the removed fields. Otherwise, troubleshooting turns into an argument about whether an error line disappeared with the secret.

The final status has three clear values: "accepted," "not accepted," or "accepted with listed untested modes." After replacing a component, issue a new report revision and repeat at least the affected phase plus the combined peak load. Do not replace a DIMM after a memory error and sign the old successful CPU test: the memory controller is inside the processor, and the work may have changed thermal contact or channel population.

A one-day test catches early defects but does not replace operations

Even a clean 24 hours cannot predict fan wear, flash degradation years later, or a firmware bug triggered by a rare production combination. Acceptance reduces the risk of early failure and incorrect assembly. Operations still needs continuous collection of SEL, SMART, ECC, temperature, and power events, scheduled storage self-tests, and redundancy checks under the site's procedure.

Extend the window where failure cost is high or a complete pass cannot physically fit. A large RAM configuration deserves several passes, a high-capacity HDD needs time for an extended self-test and full write, and GPUs or network accelerators add their own workloads. Do not speed testing by increasing temperature or disabling protection. That changes an acceptance test into a destructive experiment and may erase evidence needed for a warranty case.

A popular recommendation is to heat the server with one maximum workload overnight. It is convenient because it produces one simple graph. It is weak as an acceptance method because constant load does not test power transitions, varied compute paths, disk integrity, or redundant feed transfer. Alternating phases followed by a combined peak is more likely to expose a defect and associate it with a subsystem.

After a clean run, keep the package as a baseline for that serial number. A month later, new corrected ECC events, growing media error counts, or a change in temperature differential can be compared with this record instead of an on-call engineer's memory. If the test produces a repeatable hardware event, do not bargain with the log to protect a launch date. The day has already paid for itself: the defect appeared while replacement is still planned rather than emergency work.