You have a GPU running a training job or serving a model. The process is alive, the logs are scrolling, and nothing has crashed. That tells you almost nothing about whether the card is actually doing useful work. A GPU can sit at a fraction of its capacity for hours because the data loader is starved, the batch size is too small, or a second process is quietly holding half the memory.
On your own workstation, an underused GPU costs you some electricity. On a rented GPU, billed by the hour or by the second, it costs you money for compute you never received. The fix is not complicated: look at the card while the job runs. This post walks through the tools people actually use to do that, how to install each one, what each is best at, and a few habits worth building.
TL;DR:
nvidia-smiships with the NVIDIA driver and is the baseline every NVIDIA box has. For everyday live monitoring in a terminal,nvitopis the most complete tool.gpustatis the quickest one-line summary,nvtopis the classic htop-style dashboard, andjupyterlab-nvdashboardsuits people who work inside notebooks. If your workload runs on Aquanode, the Metrics tab shows the core numbers in the console without you installing or SSHing into anything.
Before you start
Everything below assumes three things:
- A machine with at least one NVIDIA GPU. Most of these tools read from NVIDIA's management library (NVML), so they are NVIDIA-specific. AMD cards have their own tooling (
rocm-smi,amd-smi) that is outside the scope of this post. - A working NVIDIA driver that matches the card. The driver is what installs NVML and
nvidia-smiin the first place. Ifnvidia-smidoes not run, none of the other tools will either. - A driver new enough for the CUDA version your framework needs. Each driver release supports CUDA up to a certain version, and
nvidia-smiprints that ceiling in its header. If your PyTorch or JAX build expects a newer CUDA than the driver supports, fix that first.
A recent Linux distribution is the common case, and the install commands below use Ubuntu's apt and Python's pip. On a rented instance from the marketplace the driver is typically already installed, so you can usually skip straight to the tools.
nvidia-smi
nvidia-smi (NVIDIA System Management Interface) is the tool nearly everyone reaches for first, because it is already there. It ships as part of the NVIDIA driver package, so any machine with a working driver has it. No separate install step is needed.
Basic usage
Type the command on its own:
nvidia-smi
You get a table per GPU with the driver and CUDA versions, temperature, power draw against its cap, memory used against total, and utilization, followed by the processes on each GPU and their memory. It is a snapshot, perfect for a quick "is my job even on the GPU?" check.
A live view
For a continuously updating view, wrap it in the Linux watch command:
watch -n 0.1 nvidia-smi
That reruns it every tenth of a second. The built-in -l loop flag works too, but watch redraws in place and reads better. A 1 second interval is plenty for most jobs.
Logging metrics to a file
The query interface is where nvidia-smi gets genuinely useful for longer runs. You pick exactly which fields you want and get machine-readable CSV back:
nvidia-smi --query-gpu=timestamp,name,pci.bus_id,temperature.gpu,utilization.gpu,utilization.memory --format=csv -l 1
This prints one CSV row per GPU every second. Add -f gpu_log.csv to write it to a file instead of the terminal, and you have a simple utilization log you can load into pandas or a spreadsheet after the run. The full list of queryable fields is available from nvidia-smi --help-query-gpu.
For device-level and process-level streams, nvidia-smi dmon and nvidia-smi pmon print rolling per-second stats in a compact columnar format, handy for piping into a file during a benchmark.
Checking for thermal throttling
A GPU that runs too hot slows itself down to protect the silicon, and your job just gets mysteriously slower. To see whether that is happening:
nvidia-smi -q -d TEMPERATURE,PERFORMANCE
The temperature section lists the thresholds at which the card will slow down or shut down. The performance section lists the active "clocks event reasons" (older drivers call them throttle reasons). If you repeatedly see a hardware slowdown or thermal slowdown marked active under load, cooling is the bottleneck. On your own hardware, that means better airflow, more spacing between cards, or a better chassis. On rented hardware you cannot fix the fans, so the practical move is to report it or move the workload to a different machine.
Inspecting topology
On multi-GPU machines, how the cards connect to each other matters for anything that moves data between them (tensor parallelism, data-parallel gradient syncs):
nvidia-smi topo -m
This prints a matrix showing the link between every pair of GPUs and between GPUs and CPUs: NVLink connections, a shared PCIe switch, or a path that has to cross the CPU interconnect. If two GPUs you planned to pair only talk through the host, expect slower transfers than the spec sheet suggests.
It can also change settings like power limits and compute mode (usually as root). See nvidia-smi --help for everything.
gpustat
gpustat is a small Python tool that reads the same information and prints it far more compactly: one colored line per GPU with the model, temperature, utilization, memory, and the processes on it. If nvidia-smi's table feels like too much to scan, this is the fix.
Install
pip install --user gpustat
The --user flag installs it just for your account. If you prefer to keep command-line tools isolated from your project environments, pipx install gpustat does the same job.
Usage
Just run gpustat for a one-shot summary. The most useful flag combination is:
gpustat -cup
That adds the command name (-c), the user (-u), and the PID (-p) of every process on each GPU. On a shared machine, it answers "whose job is eating GPU 3?" in one glance without reaching for a scheduler.
For a refreshing view, use its built-in watch mode:
gpustat --watch
It works like wrapping the command in watch, but keeps the colors intact, which plain watch tends to strip.
nvtop
If you have used top or htop to watch CPU and memory on Linux, nvtop will feel immediately familiar. It is a full-screen terminal dashboard for GPUs, with scrolling graphs of utilization and memory for each card and a process list underneath. It is the tool to leave running in a split pane while a job trains, because the graphs make trends obvious in a way a single number never does.
Install
On Ubuntu and Debian-based systems:
sudo apt install nvtop
It is packaged for most other major distributions too, and the project's repository documents building from source if yours lacks it.
Usage
Run nvtop and the interface comes up with live graphs at the top and processes below. You can change what the graphs show, how processes are sorted, and the refresh interval from inside the interface, and those preferences can be saved so they persist between sessions.
The standout feature is borrowed straight from htop: you can select a process in the list and kill it interactively. That is handy when a distributed training script has hung and left zombie workers holding GPU memory, or when a forgotten notebook kernel is sitting on 20GB of VRAM.
nvitop
nvitop tries to take the best of everything above and put it in one tool: the depth of information from nvidia-smi, the readable color coding of gpustat, and the live, interactive interface of nvtop. For day-to-day work, it is the one we would install first.
Install
pip install --user nvitop
As with gpustat, pipx install nvitop is a good alternative if you want it outside your project environments.
Usage
Running nvitop on its own launches monitor mode: a persistent, self-refreshing screen with per-GPU bars and history, host CPU and memory, and a process table. If you just want a single printout, closer to what nvidia-smi gives you, pass the one-shot flag:
nvitop -1
A few things make it stand out in practice. It adapts its layout to your terminal size, trimming down to the essentials in a small pane rather than wrapping into an unreadable mess. Like nvtop, it lets you sort processes and kill them interactively, and when you select a process it highlights the GPU that process is running on. That link between process and device is surprisingly useful on 8-GPU machines.
It is also a library, not just an app. nvitop exposes a Python API for querying devices and processes from your own code, and it ships callbacks for Keras and PyTorch Lightning, so you can log GPU stats alongside your training metrics without writing the plumbing yourself.
jupyterlab-nvdashboard
Not everyone lives in a terminal. If most of your work happens in JupyterLab, jupyterlab-nvdashboard (from the RAPIDS project) brings GPU monitoring into the notebook interface itself.
Install
pip install --user jupyterlab_nvdashboard
Restart JupyterLab after installing so the extension loads.
Usage
The extension adds GPU dashboards you can open from JupyterLab's interface and dock next to your notebooks: interactive charts for GPU utilization, memory, and data throughput that update as your cells run. It fits exploratory work: run a cell, watch memory climb, adjust batch size right there. It is less useful for headless training jobs with no notebook open.
Other options worth knowing
The tools above cover live, on-the-box monitoring. A few others fill different gaps:
- Weights & Biases. If you already log experiments to W&B, it records system metrics, including GPU utilization, memory, temperature, and power, alongside each run by default. You get GPU history tied to the run that produced it, with no extra setup.
- nvidia_gpu_exporter with Grafana. For teams that already run Prometheus and Grafana, this exporter wraps
nvidia-smiand publishes its metrics for scraping, so GPU stats land on the same dashboards and alerts as the rest of your infrastructure. - PyTorch Profiler and NVIDIA Nsight Systems. When you need to know why a step is slow rather than whether the GPU is busy, profilers trace individual kernels, memory copies, and CPU-GPU synchronization points. They add overhead, so use them for targeted investigation runs rather than leaving them on.
- pynvml. The Python bindings for NVML (published on PyPI as
nvidia-ml-py) let you read any GPU statistic from inside your own code. It is the building block for custom logging, early-stopping on memory pressure, or your own monitoring script when nothing off the shelf fits.
Comparison at a glance
| Tool | Type | Install | Best for |
|---|---|---|---|
| nvidia-smi | CLI | Ships with the NVIDIA driver | Quick checks, CSV logging, throttling and topology |
| gpustat | CLI | pip install --user gpustat | Compact one-line-per-GPU summary on shared machines |
| nvtop | TUI | sudo apt install nvtop | htop-style live graphs, killing stuck processes |
| nvitop | TUI and Python library | pip install --user nvitop | Day-to-day live monitoring, plus a Python API |
| jupyterlab-nvdashboard | JupyterLab extension | pip install --user jupyterlab_nvdashboard | Watching GPU usage from inside notebooks |
| Weights & Biases | Hosted service | pip install wandb | GPU history tied to each experiment run |
| nvidia_gpu_exporter | Prometheus exporter | Binary or container | GPU stats on existing Grafana dashboards |
| PyTorch Profiler / Nsight Systems | Profiler | Bundled with PyTorch / NVIDIA download | Finding exactly where step time goes |
| pynvml | Python library | pip install nvidia-ml-py | Custom logging and scripts |
Skipping the SSH step on Aquanode
Every tool above assumes you are logged into the machine and running a command there. For anything deployed on Aquanode, you do not have to. Each running deployment has a Metrics tab in the console, collected by the Aquanode agent on the box, with the Metrics option on by default when you deploy. Nothing to install.
For every GPU on the box, it shows utilization, memory used against total, temperature, power draw, and SM and memory clock speeds, refreshed every few seconds. Host CPU utilization and system memory sit alongside, which matters because a GPU idling while the CPU is pinned points at data loading, not the GPU. Each reading is shown as a current value and as a chart across the session, so you can see a training loop settle in or spot the moment throughput dropped.
One detail we care about: not every card can report every sensor. Virtualized GPUs often expose no temperature or power reading at all. When that happens, the metric reads Unavailable and the chart shows a gap, instead of a fake 0 you might mistake for a real measurement. A genuine 0% utilization still shows as zero.
It is a live view of a running deployment, not a long-term metrics store. Once the instance stops, there is no live view to show, so if you need a record beyond the session, log it yourself (the nvidia-smi CSV query above works fine) or use something like W&B.
Which tool should you use?
Short version, by situation:
- Everyone: learn
nvidia-smi. It is on every NVIDIA machine you will ever touch, and its query, throttling, and topology commands answer questions the prettier tools do not. - Day-to-day live monitoring:
nvitop. It shows the most useful information, stays readable at any terminal size, handles processes interactively, and doubles as a Python library. - Quick glance on a shared box:
gpustat -cup. - Notebook-first workflows:
jupyterlab-nvdashboard, so the charts sit right next to the code you are running. - Long-term history or team dashboards: W&B for per-experiment history, or
nvidia_gpu_exporterinto Grafana for fleet-wide views. - "Why is this step slow?": a profiler, not a monitor.
Whatever you pick, the habit matters more than the tool: glance at utilization and memory in the first few minutes of every run. A job that settles at low utilization will usually stay there, and every hour it runs that way is an hour of GPU you paid for and did not use.
If you want those numbers without SSHing in, the Metrics tab is there on every running Aquanode deployment. Compare live rates on the GPU index or rent a card from the marketplace and watch it work.
