Serving Gemma Across Two Nodes with vLLM and LeaderWorkerSet, on CPU

Wed, 28 Oct 2026 00:00:00 +0000

I wanted to try two things at once: Kubernetes’ LeaderWorkerSet (LWS), the API for running a group of pods as one unit, and vLLM’s multi-node serving. The cheapest way to exercise both is not a GPU cluster. It is two small CPU nodes, one small model, and a deliberate decision to split that model across the nodes even though it fits on one.

Vllm on Experiment, Fail, Learn, Repeat

Serving Gemma Across Two Nodes with vLLM and LeaderWorkerSet, on CPU