GPU Cluster Optimization for ML Workloads

01 The question

Kubernetes schedules GPUs between containers, but a worker node can only run a GPU workload if a GPU is physically attached to it. That binding is the problem. A GPU attached to one machine is unusable by workloads on every other machine, so if several services each need a GPU for short bursts you either buy a GPU per machine and watch most of them idle, or you crowd the workloads onto the few machines that have one.

Neither is efficient, and the waste is invisible in ordinary monitoring: each GPU looks correctly provisioned, because each is attached to something that occasionally needs it.

MSCS thesis, National University of Sciences and Technology (NUST), Islamabad, 2021 · full PDF, 36 pages

02 The framework

Pool every GPU in the datacenter and hand one out only for as long as it is being used.

1. A workload that needs GPU compute requests a number of GPUs from a central server rather than expecting one on its own node.
2. The server reserves GPUs from the pool for that request.
3. On completion the GPUs are returned to the pool, and the next request from any machine in the cluster can be served by the same physical hardware.

The consequence is that a worker node with no GPU attached to it can still run GPU workloads, and the cluster needs enough GPUs for its concurrent demand rather than one per machine that might ever need one.

How it was built

• VMware ESXi hypervisors for virtualisation, managed by vCenter across bare-metal servers, forming a private cloud.
• VMware Bitfusion to pool the GPUs across every ESXi host in the datacenter.
• A Kubernetes cluster provisioned on that datacenter, with a defined worker template.
• Bitfusion client access on the workers, and bitfusion-cli in the workloads that needed GPU compute. CUDA 11.

03 What was measured

The obvious objection is that pooling costs performance, because CUDA calls and data pointers now cross a network to reach the GPU. So the same CUDA workloads were run twice: once on a VM with a GPU attached directly, once on a VM with no GPU using the pool.

About 1.5 percent overhead. Similar behaviour held for matrixMulCUBLAS and for several proprietary algorithms run the same way.

The overhead is network latency, not GPU performance. In the pooled case all arithmetic still happens on a real GPU; what changed is that the call travelled to reach it. That distinction matters, because network capacity between hosts in a datacenter is cheap relative to buying another GPU.

So the trade is a low single-digit percentage of latency against the ability to stop buying a GPU for every machine that occasionally needs one.

04 What it did not show

Stated because a limitation you find yourself is worse than one you were told about.

• One cluster, one datacenter. Sharing across clusters, and across clouds, was named as future work rather than demonstrated.
• The workloads were CUDA samples plus some proprietary algorithms. matrixMul is compute-dense and short; a workload that streams large tensors continuously would pay the network cost differently, and that was not characterised.
• No training runs. The measurements are kernel-level, not end-to-end model training.
• The pooling mechanism is gone. VMware announced End of Availability for vSphere Bitfusion in May 2023, and support ended in May 2025. The specific implementation is not something anyone should build on today.

05 What happened to the problem since

The mechanism went away and the problem got worse. GPUs became more expensive and harder to get, and the workload mix moved toward LLM inference, which is far more bursty than the batch jobs of 2021.

The part of the thesis that aged best was not the pooling. It was the premise underneath it: you cannot allocate GPUs sensibly until you can measure what they are actually doing, and the standard utilization figure does not tell you.

That measurement problem is what truthscale addresses. nvidia-smi reports “GPU utilization” and means the fraction of time in which at least one kernel was resident, so a single small kernel on 1 of an H100’s 132 streaming multiprocessors reads 100 percent. Autoregressive decode pins that flag while the tensor cores idle. Pooling, rightsizing and autoscaling all depend on a number that means something, and the number in front of most people does not.

More on the current work in projects.