KV quantization: how many bytes, and at what error?

01 / QUESTION

Compare tensor and row-wise int8 quantization, including scale bytes and attention error under outliers.

METHOD

Freeze protocol and source before execution, save raw records, then independently recompute metrics and verify hashes.

RESULT

36 quantization comparisons including scales; maximum absolute output error is 4.136, not lossless.

OBSERVATIONS

36 quantization comparisons including scales; maximum absolute output error is 4.136, not lossless.

CONCLUSION

Conclusions are limited to the stated variables and acceptance criteria, retaining failures and unverified scope.

LIMITATIONS

Teaching-scale fixed conditions, not online model quality, production throughput or hardware safety. The complete assistant and VL01 remain unfinished.

REPRODUCE

The article provides a pinned revision, environment, runner and independent audit commands.

1. Mechanisms

   KV quantization: how many bytes, and at what error?

   Compare tensor and row-wise int8 quantization, including scale bytes and attention error under outliers.

   2026-10-079 MINPublished
2. PROJECT / 002

   Java LLM Practice

   Reproducible Java conversation, retry and persistence boundaries, alongside attention, cache-cost and quantization-error experiments.

   Building
3. Mechanisms

   Tiled attention: preserving results without a full score matrix

   Derive online softmax from a failing block-average baseline and separate smaller temporaries from measured speed.

   2026-10-079 MINPublished