The hot topic is whether GPUs really need to be so power-hungry and noisy.
Imagine the core mathematics behind language models and the transformer architecture. Each answer is produced by scanning the model and the context. There is both engineering and math behind it.
Engineering traces the problem back to the hardware blocks that do the job. Traditional processors worked on a series of data. GPUs are at their best when they can work on blocks of thousands of bits at a time. These blocks are the foundation of all AI logic. A question and its preceding context can easily be answered with a single block. This is called Retrieval-Augmented Generation (RAG). The current sentence in a generated book is returned based on the set of attributes that specify what it is. A single block is enough to respond.
Often, when AI generates content, the information is hidden across many blocks. This makes GPUs power-hungry. We need to process 10 or 10,000 blocks to get that single sentence. When it is returned, it is returned based on general attributes that are not too specific. This makes AI generative. It can be creative. It also spreads the useful information across too many hardware blocks.
As models evolved, a technique called Mixture of Experts emerged. It restricted the topics each model can respond to, reducing the number of blocks to search to a handful, let's say 100 in our example. Models like DeepSeek achieved an order of magnitude in cost savings by training these simpler models and attaching them to each other. One answers math, another healthcare. They do not mix in training.
The math is similar to data compression. In fact, all models compress an enormous, petabyte-scale data corpus into a single array of blocks. This is compression. The difference is that CPUs shortened the code words, while GPUs use something similar called arithmetic coding. Arithmetic coding uses the same block lengths, but it embeds many code words within them. Apple can reserve the range 0.0–0.1, peach 0.1–0.7, and melon 0.7–1.0.
In machine learning, blocks are arrays of fixed-point or floating-point numbers. When we search for a sentence in context, we solve linear equations with these, along with some non-linear regularization. The answers encode a sentence. The original operating systems dealt with functions packed into blocks. Eventually, reordering functions with similar use cases and stacking them together made them super fast.
Machine learning will evolve in a similar way. If 100 data blocks need to be trained into 100 model blocks, the engineering becomes simple. The algorithmic complexity is quadratic, so about 10,000 operations are needed. This is costly, even if done in parallel.
What will happen to the data centers that many people are protesting against? We will likely make training more granular and train blocks separately. Once separated, blocks can be chained together like files, as skills. Inference then only needs to scan them individually, making the memory requirement an order of magnitude smaller.
Communities will not lose out. The leftover energy can be reused to create cheaper competition for power and heating. The existing data centers are clear benchmarks. Also, some workloads, such as video, still require GPUs. Block compute was introduced for a reason, and graphics will always be the dominant workload.