quantization-impact-analyzer

Analyzes per-layer quantization sensitivity in GPT-2 and finds optimal mixed precision configurations given a memory budget.

For detailed protocol, layer classification, and full results see DESIGN.md.

---

Portfolio Context

Part of an 11-project series on LLM inference infrastructure:

Project	Focus	Key Finding
kv-cache-compaction-lab	KV-cache compaction	ThresholdCompaction dominates
prefix-cache-sim	Prefix sharing	LFU dominates; 60%+ hit rate
llm-inference-scheduler	Continuous batching	ChunkedPrefill eliminates starvation
tensor-memory-allocator	GPU tensor allocation	Free-list beats buddy/slab
llm-serving-sim	End-to-end serving	41% lower TTFT, 94% prefix hit rate
speculative-decoding-sim	Speculative decoding	6.06x max speedup
moe-router-sim	MoE routing	ExpertChoice best balance
admission-control-sim	Admission control	Tight budget maximizes goodput
kv-cache-disaggregation-sim	Prefill/decode disaggregation	Disagg wins at high load + long prompts
speculative-decoding-validation	Real GPU validation	Simulation predictions confirmed
quantization-impact-analyzer	Weight quantization sensitivity	INT8-g32: 1.8x compression, +0.13 PPL

---

Key Findings

1. Group-wise quantization is the critical enabler

INT4 per-tensor:  avg PPL delta = +108  (model collapse)
INT4-g128:        avg PPL delta = +8.89  (98% reduction)
INT4-g32:         avg PPL delta = +2.23  (99% reduction)

Group-wise quantization is not an optimization -- it is the difference between a usable and unusable quantized model.

2. INT8-g32 is the best practical default

Compression: 1.8x
Memory: 133.4 MB vs 237.1 MB FP16
PPL: 20.74 vs 20.60 baseline (delta = +0.13)

Nearly lossless. Straightforward to implement.

3. INT6-g32 is the best sub-50% memory operating point

Compression: 2.3x
Memory: 103.7 MB (44% of FP16)
PPL: 23.87 (delta = +3.27)

Best option when memory is the hard constraint.

4. mlp_proj layers dominate quantization sensitivity

Layer type    Per-tensor INT4 avg MSE    Group-wise INT4 avg MSE
mlp_proj            1083.77                     2.19
attn_qkv             128.01                     1.70
mlp_fc               117.07                     2.09
attn_proj             40.19                     3.17
pos_embed              2.55                     3.87

MLP projection layers are catastrophically sensitive to per-tensor INT4. Group-wise quantization largely equalizes sensitivity across layer types.

5. Mixed precision vs uniform: complexity may not be worth it

Policy              Compression   PPL delta
Uniform INT8-g32       1.8x         +0.13
Balanced mixed         1.5x         -0.09

The simpler uniform INT8-g32 achieves higher compression than the Balanced mixed precision policy at similar quality. Mixed precision adds value mainly near hard memory constraints.

---

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install transformers torch accelerate pandas

# Per-layer sensitivity (per-tensor)
python3 analyze_quantization.py

# Per-layer sensitivity with group-wise
python3 analyze_quantization_v2.py

# Mixed precision search (greedy)
python3 mixed_precision_search_v2.py

# Mixed precision with manual policies (final comparison)
python3 mixed_precision_search_v3.py

---

Files

analyze_quantization.py        Per-tensor sensitivity (50 layers x 3 bits)
analyze_quantization_v2.py     Group-wise sensitivity (50 layers x 9 configs)
mixed_precision_search.py      Greedy search v1 (per-tensor, for reference)
mixed_precision_search_v2.py   Greedy search v2 (group-wise configs)
mixed_precision_search_v3.py   9 policies compared (final)

results/
  per_layer_error.csv          150 rows (per-tensor)
  per_layer_error_v2.csv       450 rows (group-wise, 9 configs)
  mixed_precision_search.csv   v1 greedy results
  mixed_precision_search_v2.csv v2 greedy results
  mixed_precision_v3.csv       9 policy comparison

---

Policy Comparison Summary

Policy            Comp   Mem MB   PPL    Delta
FP16 (baseline)   1.0x   237.1   20.60   +0.00
INT8-g32          1.8x   133.4   20.74   +0.13  <- best practical
INT6-g32          2.3x   103.7   23.87   +3.27  <- best sub-50%
Balanced          1.5x   157.2   20.51   -0.09  <- near-baseline
INT5-g32          2.7x    88.9   39.15  +18.55
INT4-g32          3.2x    74.1  167.30 +146.70