LLM mechanics (7): why is caching slower even when it saves computation?

01 / QUESTION

4,860 measured queries compare three admission policies, connecting byte budgets, working sets, computation and local elapsed time.

METHOD

81 configurations, one excluded warm-up trace and five measured traces each: 4,860 measured requests and 1,944 archived arrays.

RESULT

Maximum error 4.16e-16. At 16KiB, longest-only admission halves three-scope projection rows, yet 6.723ms exceeds fresh computation at 1.697ms.

OBSERVATIONS

Source/weight signatures, copying and Python management are timed; two branches still get zero hits at 8KiB.

CONCLUSION

Admission can improve the working set, but saved computation is not measured speedup. Record correctness, bytes, work and time together.

LIMITATIONS

Tiny untrained NumPy model, no GPU or CPU affinity. Not online end-to-end latency; no component attribution or significance test.

REPRODUCE

Pinned code and raw records are in the repository; the article provides run, audit and replay commands.

1. Mechanisms

   LLM mechanics (7): why is caching slower even when it saves computation?

   4,860 measured queries compare three admission policies, connecting byte budgets, working sets, computation and local elapsed time.

   2026-10-0512 MINPublished
2. PROJECT / 002

   Java LLM Practice

   Reproducible Java conversation, retry and persistence boundaries, alongside attention, cache-cost and quantization-error experiments.

   Building
3. Mechanisms

   LLM mechanics (6): why can a larger prefix cache still get zero hits?

   324 requests across cold-start traces and 648 arrays test byte-bounded prefix checkpoints, working sets, isolation and LRU eviction.

   2026-10-0411 MINPublished
4. Mechanisms

   LLM mechanics (4): when a prefix changes, is the old KV cache still valid?

   Three seeds and six conditions compare unchecked and keyed prefix reuse; numerical equality does not establish permission to share across scopes.

   2026-10-039 MINPublished