When your company invests heavily in GPU hardware, a deep understanding becomes crucial, especially at the executive level. But navigating this terrain can be tricky.
In the past, brands like Intel and AMD provided a convenient shorthand: "Intel is reliable, AMD is compatible." This simplified decision-making and agreement. However, with GPUs, TPUs, and the concept of "tokens," the connection between hardware and pricing has become less direct. This article aims to clarify these nuances, fostering better understanding within your teams.
CPUs were once the workhorses of computing. Their rigid architecture prioritized compatibility with complex software, like the tens of million lines of code in Windows without applications from decades ago.
Think of CPUs as forklifts in a warehouse. They operated on limited memory, often in isolation. They'd grab data, process it, and then deposit the result elsewhere. This was due to the limitations of register space, requiring fast memory to be physically close to the processor core on a two-dimensional chip.
As hardware advanced, maintaining a common standard with the same set of registers became easier. Multiple layers allowed more "forklifts" to work in parallel, but bottlenecks remained, much like collisions in a busy warehouse.
The architectural limitation was that even if code could logically handle 2048 or 4096 bits, it required extensive pipelines to do so quickly. Moreover, only a few of these expensive near registers were truly active, while others sat idle.
GPUs changed the game. Originally designed for graphics and images, their registers represent pixels and are highly unified. This allows "forklifts" to work in parallel, each with its own aisle in the warehouse.
Generative AI required a bit more abstraction. When training AI, text is processed in chunks, each mapped to an area within the GPU's processing context like this word in the article to the numbers 1001-1003. These chunks and their context define a stage in the data stream. Traditionally, these were words; now, they're often syllables, known as "tokens." Tokens on the input and output are mapped to embeddings inside GPU registers, which are clusters of related numbers within the GPU context.
Because GPU contexts are unified, they can handle much greater complexity. This expressive power made generative text and understanding possible. An embedding represents a syllable's position, its paragraph and chapter context, and associated sentiments and labels. Areas then represent texts that are closely related, making them easy to search, address, and assemble into meaningful text.
AI inference is still possible with or without GPUs. The streaming architecture even allows embedded processors to answer questions, even with audio. When negotiating hardware, remember that GPUs offer a more uniform way of handling information. This makes them powerful, but the laws of physics still apply. While GPUs use their bits more evenly, they eventually face the same context limitations. The speed of DRAM is where clever reasoning and code search can make a real difference.