One question splits this whole field in two: are you using an AI model, or building one? Using a finished model to answer a question is called inference, and decentralizing it is hard but solved, running in production today. Building the model in the first place, out of mountains of examples, is called training, and decentralizing that is the frontier nobody has cracked.
Inference is a single self-contained job. A question comes in, the model runs once, an answer comes out, and you can check it afterward. Training is not one job but an enormous tightly coupled marathon, and it cannot be cleanly chopped into independent pieces. Every machine working on it has to stay in step with every other machine, constantly, or the whole effort falls apart.
To see why, we ground gradient descent from scratch: predict, measure how wrong the prediction was, nudge billions of internal numbers slightly in the direction that reduces the error, repeat. Gradient comes from the Latin gradus, a step. It is a walk downhill taken one step at a time, and in distributed training every machine has to take the same step together. That is the whole problem in one sentence.
Three obstacles follow. Synchronization, where after every step the network becomes the bottleneck long before the chips do. Compression, where sending a rougher summary buys speed at the cost of exactness, the same trade as turning down the resolution on a photograph. And heterogeneous hardware, where machines that are not identical quietly produce slightly different numbers from the same arithmetic, which is why some teams are chasing bitwise-reproducible execution.
We cover what Gensyn, Nous Research and Prime Intellect are each betting on, and end on the honest state of it: nobody yet knows whether very large decentralized pre-training is achievable, or whether the gravity of putting all the machines in one building wins.
This is Basics 5 of 8, the start-here arc. Each episode stands on its own and assumes nothing.
The voices in this show are AI-generated; the research and writing are human. Decentralized AI, layer by layer.