Serving Gemma Across Two Nodes with vLLM and LeaderWorkerSet, on CPU
··4060 words·20 mins
I wanted to try two things at once: Kubernetes’ LeaderWorkerSet (LWS), the API for running a group of pods as one unit, and vLLM’s multi-node serving. The cheapest way to exercise both is not a GPU cluster. It is two small CPU nodes, one small model, and a deliberate decision to split that model across the nodes even though it fits on one.