Fitting a 12B Model into Two Small CPU Nodes: Gemma 4 on Graviton with vLLM

Mon, 02 Nov 2026 00:00:00 +0000

In the last post I argued that you should not split a model that fits on one machine: the 1B model I benchmarked was faster, steadier and cheaper in a single pod. The flip side of that argument is the case where splitting is not a choice. When the weights do not fit, pipeline parallelism is how you run the model at all.

Memory on Experiment, Fail, Learn, Repeat

Fitting a 12B Model into Two Small CPU Nodes: Gemma 4 on Graviton with vLLM