KV quantization axes: why Key and Value need separate reasoning

01 / QUESTION

162 archived INT8 comparisons examine tensor, token and K-channel/V-token scales, including cases where the hybrid loses.

METHOD

Freeze source and protocol, execute local comparisons, archive raw records and independently recompute with hash checks.

RESULT

Across 162 comparisons, the hybrid averages 1.9215% KV output error with K-channel outliers; token-wise is better on clean inputs.

OBSERVATIONS

Across 162 comparisons, the hybrid averages 1.9215% KV output error with K-channel outliers; token-wise is better on clean inputs.

CONCLUSION

Choose quantization axes from outlier structure and independently recompute output error and scale overhead.

LIMITATIONS

Synthetic float32 arrays and CPU NumPy; no trained model, GPU speed, peak-memory or task-quality claims.

REPRODUCE

The article supplies a pinned commit, locked environment, separate output directory and audit command.

1. Mechanisms

   KV quantization axes: why Key and Value need separate reasoning

   162 archived INT8 comparisons examine tensor, token and K-channel/V-token scales, including cases where the hybrid loses.

   2026-10-0910 MINPublished
2. PROJECT / 002

   Java LLM Practice

   Reproducible Java conversation, retry and persistence boundaries, alongside attention, cache-cost and quantization-error experiments.

   Building
3. Mechanisms

   KV quantization: how many bytes, and at what error?

   Compare tensor and row-wise int8 quantization, including scale bytes and attention error under outliers.

   2026-10-079 MINPublished