How to Run a GPU Stress Test on Linux (Plus CPU and I/O)

You just rented a GPU box. Before you point a twelve-hour training run or a production inference server at it, spend a few minutes making it sweat. A short GPU stress test tells you three things a clean nvidia-smi readout never will: whether the hardware holds up under sustained load, whether the cooling keeps the card out of thermal throttling, and whether the driver stack actually behaves when every core is busy.

TLDR Install stress to load the CPU, memory and disk, and build gpu-burn to load every GPU at once. Run both together for a minute or more while htop, iotop, sensors and nvidia-smi watch from split tmux panes. A healthy box finishes the full run with no OOM kills, no errors from gpu-burn, and no active thermal slowdown in nvidia-smi -q.

This guide assumes the NVIDIA driver and CUDA toolkit are already present, which is standard on CUDA base images. Commands are for Ubuntu.

Install stress, htop, iotop and lm-sensors

Everything on the CPU side comes from the standard Ubuntu repositories:

sudo apt-get update
sudo apt-get install -y stress htop iotop lm-sensors

Then start a load that touches every part of the host at once:

stress --cpu `nproc` --vm `nproc` --vm-bytes 1GB --io `nproc` --hdd `nproc` --hdd-bytes 1GB --timeout 60s

Each flag spawns a group of workers, one per core here thanks to nproc:

• --cpu workers spin on square root math, pinning every core.
• --vm workers repeatedly malloc and free memory, with --vm-bytes setting how much each one grabs.
• --io workers hammer sync(), forcing buffered writes out to disk.
• --hdd workers write files and unlink them, with --hdd-bytes controlling the size per worker.
• --timeout stops everything after the given time.

Watch the memory math. nproc VM workers at 1GB each on a 64-core box asks for 64GB. If the host has less free RAM than that, drop --vm-bytes or the worker count, or you will be testing the OOM killer instead of the machine.

Install and run gpu-burn

gpu-burn is a small, widely used open source tool that runs heavy matrix multiplies on every visible GPU and checks the results for errors. Build it from source:

git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
make
./gpu_burn 60

The argument is the run time in seconds. Sixty is fine for a smoke test; for a box you plan to keep for days, run 10 to 15 minutes so the card reaches a steady temperature. Add -d to test double precision instead of single.

If make fails with nvcc: not found, the CUDA toolkit is missing or not on your PATH. Point the build at it with make CUDAPATH=/usr/local/cuda.

While it runs, gpu-burn prints progress and an error count per GPU. When it finishes, every GPU should report OK. Any GPU marked FAULTY is producing wrong answers under load, and that card should not get your job.

Monitor with htop, iotop, sensors and nvidia-smi

The point of running both stress tools together is to watch the whole system at once. The simplest way is tmux with four panes:

tmux new -s burn
# Ctrl-b %  splits vertically, Ctrl-b "  splits horizontally
htop
sudo iotop
watch sudo sensors
watch nvidia-smi

Run one command per pane, then start stress and gpu_burn in a fifth window. You now see CPU and memory load in htop, disk bandwidth in iotop, CPU temperatures in sensors, and GPU utilization, memory, power draw and temperature in nvidia-smi, all refreshing live.

What a healthy result looks like

A good box gets through the full run without drama:

• stress and gpu_burn both exit on their own timers. Nothing gets killed, and dmesg shows no OOM or Xid errors.
• gpu-burn reports OK for every GPU with zero errors.
• GPU utilization sits near 100% and temperatures level off instead of climbing until the card shuts down.
• nvidia-smi shows no active thermal slowdown.

Check that last point directly while the burn is running:

nvidia-smi -q -d TEMPERATURE,PERFORMANCE

Under the clock throttle (or clocks event) reasons, HW Slowdown, HW Thermal Slowdown and SW Thermal Slowdown should all read Not Active. SW Power Cap going active under a full burn is normal, since the card is simply hitting its power limit.

Signs worth stopping for: a GPU marked FAULTY, Xid errors in dmesg, a GPU dropping off nvidia-smi mid-run, thermal slowdown turning active, or clocks falling well below their usual boost range. On a rented box, the fix is easy: release it and rent another one rather than debugging someone else's hardware.

Run this once by hand on any new box. For ongoing monitoring after that, Aquanode's GPU metrics dashboard shows live utilization and temperature without keeping a tmux pane open.

Ready to test one yourself? Browse the marketplace, launch a pod, and run the burn before your first real job.