Eks on Experiment, Fail, Learn, Repeat

Fitting a 12B Model into Two Small CPU Nodes: Gemma 4 on Graviton with vLLM

Mon, 02 Nov 2026 00:00:00 +0000

In the last post I argued that you should not split a model that fits on one machine: the 1B model I benchmarked was faster, steadier and cheaper in a single pod. The flip side of that argument is the case where splitting is not a choice. When the weights do not fit, pipeline parallelism is how you run the model at all.

Don't Split a Model That Fits: One Pod vs Two for Gemma on CPU

Sun, 01 Nov 2026 00:00:00 +0000

In the last post I split a 1B-parameter Gemma model across two CPU nodes on purpose, to learn how LeaderWorkerSet and vLLM’s multi-node mode behave. I said then that splitting a model that fits on one machine makes it slower, and I listed “compare the split against a single replica” among the things I had not done.

Serving Gemma Across Two Nodes with vLLM and LeaderWorkerSet, on CPU

Wed, 28 Oct 2026 00:00:00 +0000

I wanted to try two things at once: Kubernetes’ LeaderWorkerSet (LWS), the API for running a group of pods as one unit, and vLLM’s multi-node serving. The cheapest way to exercise both is not a GPU cluster. It is two small CPU nodes, one small model, and a deliberate decision to split that model across the nodes even though it fits on one.