Fitting a 12B Model into Two Small CPU Nodes: Gemma 4 on Graviton with vLLM
·2669 words·13 mins
In the last post I argued that you should not split a model that fits on one machine: the 1B model I benchmarked was faster, steadier and cheaper in a single pod. The flip side of that argument is the case where splitting is not a choice. When the weights do not fit, pipeline parallelism is how you run the model at all.