LLM mechanics (6): why can a larger prefix cache still get zero hits?

01 / QUESTION

324 requests across cold-start traces and 648 arrays test byte-bounded prefix checkpoints, working sets, isolation and LRU eviction.

METHOD

Three seeds × three budgets × three traces of twelve requests, starting cold and storing full-prefix checkpoints.

RESULT

324 requests and 648 arrays; maximum error about 4.16e-16. The 8KiB alternating trace has zero hits, 22 evictions and no projection savings.

OBSERVATIONS

The 8KiB hot trace uses 336 projected rows versus 864 fresh; the same budget cannot hold all checkpoints for two branches.

CONCLUSION

Cache utility depends on admission and working set; evaluate capacity, reuse distance and copying together.

LIMITATIONS

Untrained NumPy model; persistent array payload only, not RSS. No GPU, production model or end-to-end timing.

REPRODUCE

Pinned code and raw records are in the repository; the article provides run, audit and replay commands.

1. Mechanisms

   LLM mechanics (6): why can a larger prefix cache still get zero hits?

   324 requests across cold-start traces and 648 arrays test byte-bounded prefix checkpoints, working sets, isolation and LRU eviction.

   2026-10-0411 MINPublished
2. PROJECT / 002

   Java LLM Practice

   Reproducible Java conversation, retry and persistence boundaries, alongside attention, cache-cost and quantization-error experiments.

   Building
3. Mechanisms

   LLM mechanics (5): after editing history, can we recompute only the changed suffix?

   27 fixed conditions and 81 raw arrays test longest-prefix cropping, continuation positions, sliding-window fallback and scope isolation.

   2026-10-0310 MINPublished
4. Mechanisms

   LLM mechanics (4): when a prefix changes, is the old KV cache still valid?

   Three seeds and six conditions compare unchecked and keyed prefix reuse; numerical equality does not establish permission to share across scopes.

   2026-10-039 MINPublished