Computer Science

A bit is the smallest piece of information: a yes or a no, written 1 or 0. Claude Shannon showed in 1948 that any message, a picture, a song, this sentence, can be measured and sent as bits.

A transistor is a switch with no moving parts. A small voltage on one terminal lets current flow between the other two, or stops it. A modern chip holds tens of billions of them, each only a few tens of atoms across in its smallest features.

Wire a few transistors together and you get a logic gate: AND outputs 1 only if both inputs are 1, OR if either is, NOT flips its input. One gate, NAND (not-and), is enough on its own to build every other gate, and so every circuit. Chain gates into an adder and the machine can do arithmetic; add a loop that remembers its output and it can store a bit. Arithmetic plus memory is a computer.

In 1936 Alan Turing described a machine that could carry out any calculation that can be written as steps, if you gave it the steps as data. In 1945 John von Neumann's report on the EDVAC laid out the practical form almost every computer still follows: the program and the data live in the same memory, and a processor repeatedly fetches an instruction, decodes what it means, executes it, and moves on.

A clock keeps all of this in step. Each tick, signals move one stage further through the circuits. A 4 GHz processor ticks four billion times a second, so one tick lasts a quarter of a nanosecond, in which light itself travels only about 7.5 centimetres.

Since the 1980s processors have become faster much more quickly than memory has. A processor can now do hundreds of operations in the time it takes to fetch one number from main memory. This gap is called the memory wall, and almost every design decision in the three chips below is a different answer to it.

The usual answer is a ladder of memories: tiny and fast close to the arithmetic, huge and slow far away. The lab below shows the ladder with times stretched so that one L1 cache read takes one second.

A core is one complete processor. A phone or laptop has roughly 6 to 16, a large server chip can have well over a hundred. Each core is big and clever, because most of its transistors are spent not on arithmetic but on guessing and organising.

Pipelining splits each instruction into stages, like a car factory line, so that a new instruction can start every tick while earlier ones finish. Superscalar cores run several pipelines side by side. Out-of-order execution looks ahead in the program and runs any instruction whose inputs are ready, then puts the results back in order. Branch prediction guesses which way each if-statement will go before it is known, and is right well over 95% of the time on typical code; a wrong guess throws away the work and costs a dozen or more ticks.

Caches (L1, L2, L3) keep recently used data close by, because programs tend to reuse the same data and to read neighbouring addresses. SIMD units (on x86 called SSE and AVX, on Arm called NEON and SVE) apply one instruction to a short row of numbers at once, a small taste of what a GPU does at huge scale.

All that machinery serves one goal: low latency, meaning the time until a single task is done. It is why a CPU is good at code full of decisions, such as compilers, web servers, games logic and spreadsheets, where the next step depends on the last.

The cost is that only a small share of the chip actually does arithmetic. When the work is a million identical, independent sums, a few brilliant workers are the wrong tool, and the limit is set by how much of the job can be split up.

An NVIDIA GPU is made of streaming multiprocessors. Each runs threads in groups of 32, called warps, that execute the same instruction at the same moment on different numbers. This is SIMT: single instruction, multiple threads. AMD's equivalent group is the wavefront. A large GPU keeps well over a hundred thousand threads in flight at once.

GPUs barely try to guess or reorder. Instead, when one warp is waiting for memory, the hardware simply switches to another warp that is ready, in a single tick. With enough work queued, the arithmetic units never sit idle. If threads in one warp take different sides of an if-statement, the two sides run one after the other with half the lanes switched off, which is why branchy code runs poorly on a GPU.

The result is throughput rather than latency: each individual task finishes no sooner, often later, but vastly more tasks finish per second. The race below shows the difference.

In 2007 NVIDIA released CUDA, which let programmers use the GPU for any parallel arithmetic, not just graphics. In 2012 a neural network called AlexNet, trained on two consumer NVIDIA GTX 580 graphics cards, won the ImageNet image-recognition contest by a wide margin, and deep learning moved onto GPUs almost overnight.

Neural networks are mostly matrix multiplication, so from the Volta generation in 2017 NVIDIA added tensor cores: units that multiply and add whole small blocks of a matrix in one instruction, in reduced precision such as 16-bit or 8-bit numbers. Feeding them needs enormous memory bandwidth, so data-centre GPUs sit beside stacks of high-bandwidth memory (HBM): an H100 reads about 3.35 terabytes per second, roughly forty times a typical laptop's main memory. Fast links such as NVLink tie many GPUs into one machine.

In 1978 H. T. Kung and Charles Leiserson proposed the systolic array, named after the heartbeat that pumps blood: a grid of tiny cells, each of which, every tick, multiplies two numbers, adds the result to a running total and passes data to its neighbour. Data is read from memory once and then reused as it flows across the whole grid, instead of being fetched again for every multiplication.

In a TPU's matrix unit the weights of the network are loaded into the grid and stay put. The inputs flow in from the left, one diagonal wave per tick, and the partial sums flow downward, picking up one more product at each cell, until finished answers drop out of the bottom. The first TPU's grid was 256 × 256, which is 65,536 multiply-adds every tick. Step through a small one below.

Google had the first TPU running in its data centres in 2015 and announced it in 2016. It used 8-bit integers and only ran trained networks. Later generations train them too, with several 128 × 128 matrix units per chip and a number format Google Brain devised called bfloat16, which keeps the range of a 32-bit float (8 exponent bits) but only 7 bits of precision. Neural networks tolerate the lost precision; the chip saves half the memory and far more than half the silicon per multiplier.

With no graphics hardware, little branch logic and a simpler cache, more of the chip goes on arithmetic and less energy is spent moving data. TPUs are joined into pods by their own chip-to-chip links: a TPU v4 pod connects 4,096 chips. The trade-off is flexibility. A TPU is superb at large, regular matrix work, mainly through Google's XLA compiler (used by JAX, TensorFlow and PyTorch), and poor at anything else.

The same idea now sits in phones and laptops as the neural processing unit (NPU), such as Apple's Neural Engine, first shipped in 2017.

The model does not see letters or words but tokens: common chunks of text found by an algorithm such as byte-pair encoding. 'the' is one token, 'unbelievable' might be three. In English a token averages about three-quarters of a word. A vocabulary has tens of thousands to a couple of hundred thousand tokens, each with an ID number.

Each token ID is swapped for an embedding: a list of a few thousand numbers (12,288 in GPT-3) that the model learns during training. Tokens used in similar ways end up with similar lists, so meaning becomes a direction in a space of thousands of dimensions. Information about each token's position in the text is mixed in, today usually by rotating pairs of those numbers by an angle that depends on the position (rotary embeddings).

The Transformer, introduced by a Google team in the 2017 paper 'Attention Is All You Need', is the design behind every major LLM. Its key step is attention. Each token makes three vectors from its embedding: a query (what am I looking for?), a key (what do I contain?) and a value (what will I pass on?).

Each token compares its query with the keys of every earlier token. Close matches get high scores, softmax turns the scores into weights that add up to 1, and the token takes the weighted blend of their values. That is how 'it' in 'the animal didn't cross the street because it was too tired' can pull in information from 'animal'. In a language model each token may look only backward, never at words not yet written.

A model runs many attention heads side by side (96 in GPT-3), each free to learn a different kind of relationship. Try a toy version below.

After attention, each token's vector goes through a feed-forward network: multiply by a large matrix, apply a simple bend (a nonlinearity such as GELU or SwiGLU), multiply by another matrix. Attention moves information between tokens; the feed-forward step works on each token by itself, and is where much of what the model has learnt about the world appears to be stored.

Attention plus feed-forward makes one layer, or block. Each block adds its output to the vector it received rather than replacing it (a residual connection), so the vector is a running stream that every layer reads from and writes to. Normalisation keeps its numbers in a sensible range. GPT-3 stacks 96 blocks. Almost every operation in a block is a matrix multiplication, which is why this software and the chips above fit together so well.

Some recent models use a mixture of experts: many feed-forward networks per layer, with a small router sending each token to only a few of them, so a model can hold many parameters but use only a fraction for each token.

At the top of the stack, the last token's vector is multiplied by one more matrix to give a score, called a logit, for every token in the vocabulary. Softmax turns the scores into probabilities. The model then picks one, appends it to the text, and runs again for the next. Generating a 500-token answer means running the whole network 500 times.

Temperature controls the pick. Near zero, the model almost always takes the most likely token and becomes repetitive and safe; higher, it takes more chances and becomes more varied, and more often wrong. Play with it below.

A new model starts with random numbers in all its matrices, its parameters, and produces nonsense. Pre-training shows it a vast amount of text, trillions of tokens of books, websites and code, and at every position asks it to predict the next token. The loss measures how surprised it was by the real answer.

Backpropagation, popularised by Rumelhart, Hinton and Williams in 1986, works out how much each parameter contributed to the error, all of them at once, by running the chain rule of calculus backwards through the network. Gradient descent then nudges every parameter a little in the direction that would have lowered the loss. Repeat that over trillions of tokens and the parameters come to encode grammar, facts, and patterns of reasoning, because those are what make text predictable.

A pre-trained model only continues text. To make it a helpful assistant it is fine-tuned on examples of good answers, then trained further with human or automated preferences (reinforcement learning from human feedback, popularised by OpenAI's InstructGPT in 2022), and, in newer reasoning models, rewarded for working through problems step by step and reaching checkable right answers.

Answering a prompt has two phases. In prefill the model reads your whole prompt at once; all its tokens go through each layer together, so the work is big matrix multiplications and the chip's arithmetic is the limit. In decode it writes the answer one token at a time, and to make each token it must read every one of its parameters from memory again. With one user, the chip's arithmetic mostly waits and the speed is set by memory bandwidth: the roofline above, at low intensity.

To avoid redoing work, the keys and values of every earlier token are kept in a KV cache, which grows with the conversation and is why long contexts cost memory. Serving many users at once (batching) lets one read of the weights serve many tokens, raising intensity. Quantisation stores weights in 8 or 4 bits instead of 16, so there is less to read. Models too big for one chip are split across many: by layer (pipeline parallelism) or by slicing each matrix (tensor parallelism). Estimate the speed limit yourself below.

This is also why a model can be confidently wrong. It produces the most plausible continuation, not a checked fact, and it knows only what was in its training data and what is in the text in front of it.