1 — From pixels to compute

Requirements: nothing but curiosity. This chapter has no experiments. The rest of the track assumes you can read C.

Every abstraction in PyTorch eventually lands on something physical: a register file with a fixed capacity, an SRAM bank that serves one request at a time, a memory bus of a certain width. Models are bound by those limits, and you can’t reason about them from Python. So this track starts at the silicon and works up, the same way the Linux tracks do.

It starts, oddly, with pixels.

The graphics pipeline

Early GPUs were built around one job, the 3D graphics pipeline:

Vertex processing → rasterization → fragment (pixel) processing → framebuffer blending

Each stage was fixed hardware. Programmability arrived in 2001, when the GeForce 3 introduced programmable vertex processing (Lindholm et al.). Fragment stages soon followed.

Researchers then noticed something. An array of fragment processors computing a lighting equation across millions of pixels does the same arithmetic as an element-wise operation across a large matrix.

The hardware was a parallel computer. It just didn’t know it yet.

The shader hack

To do general-purpose work on a GPU in the early 2000s, you had to disguise your math as a picture. Tools like BrookGPU (SIGGRAPH 2004), Cg and raw OpenGL or DirectX all worked around the same model:

1. Arrays became textures. Input floats were packed into 2D textures, using formats such as GL_FLOAT_RGBA32_NV (NV_float_buffer).
2. Loops became polygons. To run a kernel over an $N \times M$ array, the CPU drew an $N \times M$ rectangle made of two triangles.
3. Kernels became fragment shaders. The rasterizer generated one fragment per pixel. The shader did its math through texture lookups and wrote the answer as a “color” to an off-screen buffer: a pbuffer at first, and later a framebuffer object, which was approved in January 2005 (EXT_framebuffer_object).
4. Scatter was missing. A fragment shader could only write to the $(x, y)$ position the rasterizer gave it. It could gather from anywhere, but it could not do A[i] = val at an arbitrary address. Vertex processors could scatter, but with limited use (Owens et al.).

Step 4 matters most. Much of ordinary programming is writing to a computed address, and on a fragment shader you simply couldn’t. People did real science on this hardware anyway, which tells you how much the throughput was worth.

G80 and the unified shader

Why did vertex and pixel processors merge? The first reason was load balancing, not compute. Games use vertex and pixel work in varying ratios, so fixed pools sit partly idle (Lindholm et al.). One pool of general processors solves that, and it also happens to make a useful general-purpose machine.

NVIDIA wasn’t first to unify. ATI’s Xenos, in the Xbox 360, was already unified by 2005 (Doggett). NVIDIA’s GeForce 8800 GTX (G80) brought it to PC graphics, alongside CUDA, in November 2006. A public beta of CUDA reached developers in February 2007, and version 1.0 followed in June 2007 (1.0 release notes).

G80 had 128 scalar processor cores, grouped into 16 streaming multiprocessors (SMs) and 8 texture/processor clusters, plus 16 KB of shared memory and barrier instructions (Lindholm et al.). That gave programmers what the shader hack never had: read and write pointers, synchronization, and fast on-chip scratch space.

What to take with you

• A GPU is a throughput machine that grew out of graphics. Many of its odd features started as graphics features.
• Unification came from load balancing, and compute benefited.
• The SM is still the unit that matters twenty years later, so that’s where the next chapter goes.