What does it mean to be compute-bound?

A kernel is compute-bound when its throughput is capped by the arithmetic bandwidth of the CUDA Cores or Tensor Cores doing the work, rather than by how fast data can be fetched from memory. In roofline model terms, it's a kernel whose arithmetic intensity is high enough to sit under the flat compute roof instead of the slanted memory one, meaning the arithmetic pipes, not memory bandwidth, are the limiting resource.

Strictly, compute-boundedness is a property of one kernel at a time, but it's common (with a little squinting) to describe a whole workload that way when most of its kernels land in the same regime. Large diffusion model inference is generally compute-bound, and so is LLM inference during prefill, when an entire prompt is processed at once and each model weight, loaded into shared memory a single time, gets reused across every token in the batch.

A rough estimate makes the batch-size requirement concrete. Take a 500-billion-parameter model stored in 16-bit precision, about 1 TB of weights, and note it costs roughly one multiply and one accumulate per parameter per token, so around one trillion floating-point operations per batch element. On a GPU delivering one petaFLOP/s of 16-bit arithmetic bandwidth, that's a floor of one millisecond of compute per batch element. But being genuinely compute-bound at batch size one would require reading that same 1 TB of weights in that same millisecond, a memory bandwidth of 1 PB/s, far beyond anything on the market; real GPUs sit in the TB/s range. That gap is exactly why compute-bound LLM serving generally needs hundreds of concurrent requests batched together, not one, before arithmetic intensity climbs high enough to actually saturate the compute roof instead of stalling on memory.

Aquanode's GPU recommender factors this batch-size threshold into which instance type it suggests for a given model and target throughput.