Quantizationthrough the lens of CS61C

So far, we have seen fixed-point binary representations and IEEE 754 floating-point representations. As a quick review, a simple 6-bit representation in binary point for positive numbers is:

Floating point lets us store an approximation of a real number using binary scientific notation. Here I’m focusing on FP32, the 32-bit format we work with in CS61C.

In CS61C, we teach the 32-bit IEEE 754 single-precision format:

We’ll use this layout in the next chapter to understand the tradeoff between range and precision.

Given a fixed number of bits, we must decide how to divide them among the sign, exponent, and fraction. That decision affects four properties: range, precision, storage, and hardware execution.

01 / Range

The span of magnitudes the format can represent.

02 / Precision

How closely we can distinguish nearby values.

03 / Storage

How many bytes every value occupies.

04 / Execution

Hardware instructions that can process the value efficiently.

These choices have become very visible in machine learning. Modern models can contain millions or billions of parameters. Now if we follow the standard taught in 61C, then each numerical parameter occupies 4 bytes (32 bits).

With one billion parameters, changing the representation from 32 bits to 16 or 8 bits can save gigabytes of storage. We’ll calculate this more carefully in Chapter 4.

The same value can be represented differently depending on the format. Here is a preview using 0.93; Chapter 3 explains how each conversion works.

Consider 0.93. Its FP32 representation is

0 | 01111110 | 11011100001010001111011
s | exponent | fraction

This stored FP32 exponent is 126. Subtract bias of 127; the unbiased exponent is now 126 − 127 = −1.

FP16 must now represent the same exponent using its bias of 15: −1 + 15 = 14. 14 in five bits is 01110.

FP16 stores only 10 fraction bits instead of FP32’s 23. The fraction is shortened and rounded to 1101110001.

The resulting FP16 representation is:

0 | 01110    | 1101110001
s | exponent | fraction

This bit pattern now represents 0.93017578125. This conversion maps the FP32 approximation of 0.93 to the nearest available FP16 value. Next, let’s see how integer quantization works differently.

INT8 representation is different. Before applying a scale, adjacent representable INT8 values differ by 1 (−2, −1, 0, 1, 2, …). Signed INT8 stores integers from −128 to 127.

However, neural network weights often look like 0.037, −0.62, or of course 0.93 lol. So if hardware only supports INT8, how can an integer format store them?

We can’t just convert small weights to integers, that would just destroy most of the information: 0.93 → 1, 0.26 → 0, −0.26 → 0. Many different values will just collapse to a small integer.

The key idea here is an integer does not have to mean “one whole unit.”

If you have a scale s, then one integer step can instead represent a smaller real-valued interval.

Here, I assume symmetric quantization over approximately [−1, 1], using integer codes from −127 to 127. To map these endpoints, we’ll set the scale s to

Now we can interpret integer 1 as approximately 0.00787, 10 as 0.0787, 127 as 1.0.

To store 0.93, the q representation for INT8 would be

And to reconstruct we do

With floating point, each value gets its own exponent. In this INT8 example, a group of values shares a scale instead. We save space, but we still have to store that scale, and we don’t get exactly 0.93 back.

Given an original weight w and scale s, we can calculate the integer to store as

For this example, zero stays zero, and I’m assuming the rounded code fits in our integer range. If it falls outside that range, we need to clip it. You may also see quantization formulas with an extra offset called a zero-point; I’m leaving that at 0 here.

Here, w is the original value, s is the scale, and q is the integer we store. To get an approximation of w back:

Reconstruction opens up one issue, which is quantization error: the difference between an original value and its reconstructed approximation, where

For a simple example, choose s = 0.1 with zero-point 0, and round to the nearest integer code:

Quantization loses information because multiple original values can map to the same stored code. The scale determines the spacing between representable values and the range they cover: smaller steps can reduce rounding error, but cover a narrower range for a fixed integer width.

        Input features →
w₁₁  w₁₂  w₁₃  w₁₄
w₂₁  w₂₂  w₂₃  w₂₄
w₃₁  w₃₂  w₃₃  w₃₄
w₄₁  w₄₂  w₄₃  w₄₄

A scale can be applied at different levels of scale granularity:

Per-tensor scaling

Per-channel scaling

We can share a scale down a column (an input channel) or across a row (an output channel). Many weight-quantization kernels use one scale per output channel. Which layout you can actually use depends on the runtime and hardware.

Group-wise scaling

Tile-wise scaling

Smaller groups let us choose a scale that better fits the values in each group. That can reduce error, but now we have more scales to store and apply.

• Larger groups → less metadata, potentially more error.
• Smaller groups → more metadata, potentially less error.

During inference, three major categories are commonly quantized:

1. Weights: learned parameters stored with the model.
2. Activations: temporary intermediate tensors produced while processing the current input.
3. KV cache: attention keys and values retained from earlier tokens so they do not need to be recomputed.

KV-cache memory grows with sequence length, batch size, and number of layers; it also depends on the model’s attention dimensions and the cache datatype.

Weight and activation precision

Quantized models are often described using W_xA_y notation, where W gives the weight bit width and A gives the activation bit width. Using different formats for different model components is a form of mixed-precision inference.

For example, in W8A16, weights can use INT8 to reduce storage and memory traffic, while activations may remain FP16 or BF16 because their input-dependent ranges can make low-bit integer quantization more difficult. Activations can also be quantized, as in the W8A8 example below.

The notation alone doesn’t specify the specific format; it only provides the bit width (8-bit, 16-bit, etc.).

So a system could use:

• Weights: INT8
• Activations: FP16
• KV cache: FP8
• Selected sensitive operations or accumulations: higher precision

The runtime must be able to use the representation efficiently. Some costs include:

• Unpacking and dequantization: extracting packed low-bit values and reconstructing their approximate values when needed.
• Scale metadata and application: storing, loading, and applying shared scales.
• Datatype conversions: converting between the storage format and the format used for computation.
• Unsupported low-precision instructions: hardware may lack native operations for the chosen format.
• Inefficient kernels: an implementation may fail to use the hardware efficiently.
• Padding and alignment: extra storage or work may be needed to satisfy hardware-layout requirements.
• Higher-precision fallbacks: some operations may run in FP16, BF16, or FP32 instead.

This is the part I find interesting: making the model smaller doesn’t finish the job. We still have to check whether moving fewer bytes and using lower-precision arithmetic saves more time than unpacking, scaling, and conversion take.

I wrote and drew these notes by hand on my iPad. Here are compressed copies of the original handwritten PDFs:

• Part 1 · 14 pages · PDF, 3.6 MB
• Part 2 · 4 pages · PDF, 1.1 MB