Don't Split a Model That Fits: One Pod vs Two for Gemma on CPU
··2610 words·13 mins
In the last post I split a 1B-parameter Gemma model across two CPU nodes on purpose, to learn how LeaderWorkerSet and vLLM’s multi-node mode behave. I said then that splitting a model that fits on one machine makes it slower, and I listed “compare the split against a single replica” among the things I had not done.