Library of human understanding

TPU

Specialized silicon designed for tensor operations at scale.

A tensor processing unit (TPU) is an application-specific integrated circuit (ASIC) designed by Google to accelerate machine learning workloads. Unlike a CPU, which must be general-purpose, or a GPU, which trades some generality for parallelism, a TPU is built for one job: multiply large matrices fast. By removing the flexibility, Google engineered a chip that delivers more matrix operations per watt and per dollar than any GPU.

TPUs shifted the economics of AI training. A massive neural network that would take weeks on CPUs and days on GPUs can be trained in hours on TPUs. Google uses TPUs in its data centers for search ranking, translation, image recognition, and language models. The tradeoff is that a TPU is useless for most other tasks: it cannot run a web server or encode video. But for deep learning at scale, it is unmatched.

Application-Specific Design

A TPU is an ASIC: a custom chip designed for one application. Unlike a GPU, which is flexible enough to run any parallel code, a TPU is optimized specifically for matrix multiplication with low precision (8-bit or 16-bit) arithmetic. It has no branching, no dynamic memory access, no interrupts—all the features that make a general-purpose processor flexible but slow.

This specialization is ruthless. A TPU has massive arrays of multiply-accumulate units (MACs) that do nothing but compute A×B+C. It has memories optimized for the specific access patterns of matrix multiplication. It has interconnects tuned for moving data between these arrays. The result is 100–1000× higher throughput than a CPU for neural networks, at the cost of being useless for anything else.

Systolic Arrays for Tensor Operations

A TPU uses a systolic array: a grid of processing elements connected in a regular pattern, where data flows through like blood through arteries. Each processing element computes one product and passes it to the next, creating a pipeline of multiplication. The name comes from biology: data pulses through the array rhythmically.

Systolic arrays are ideal for matrix multiplication. To compute C = A × B, you can feed A through the array row by row, feed B column by column, and collect the results as they flow out. The throughput is enormous because the array is always busy: there is no idle time waiting for memory. All data moves in lockstep.

Specialized Variants for Training and Inference

Google has released different TPU versions for different purposes. Training TPUs use higher precision (16-bit or 32-bit) arithmetic to avoid numerical instability when backpropagating gradients. Inference TPUs use lower precision (8-bit), consuming less power and fitting more models on a chip. Edge TPUs, embedded in smartphones and IoT devices, are scaled-down versions designed for battery-powered operation.

The most recent TPUs, like the TPUv6, have hundreds of billions of transistors and can achieve petaflop-scale performance on neural network operations. At this scale, the bottleneck shifts: neural networks are limited not by compute but by memory bandwidth—the rate at which data can flow from memory to the compute units. Google's designs address this by carefully optimizing the memory hierarchy and the layout of the data.

The people

Norman Jouppi

born 1960 · Computer Science, Hardware Design

Led the design of Google's Tensor Processing Unit (TPU) and subsequent generations, demonstrating the value of application-specific processors for machine learning.
Pioneered the concept of designing chips specifically for AI workloads, shifting the industry toward specialized accelerators.
An influential researcher in computer architecture and the design of specialized processors.

“To be a computer architect, you have to understand the application domain so well that you can throw away most of the machine and still solve the problem.”

Research paper and talks on TPU design

Quotations appear only when we could check them against a named source. Sayings we could not trace are kept out.