LLM mechanics (7): why is caching slower even when it saves computation?

Pinned code · Raw evidence and audit · Experiment record

The previous study found an 8KiB cache repeatedly evicting checkpoints without a single hit. Saving fewer checkpoints seems like an obvious improvement. This time we measure both work and elapsed query time: admission improves some traces, yet the tiny NumPy model remains faster without caching.

Separate three questions

Question	Check	Supported conclusion
Are the results equivalent?	Compare cached and full arrays for the same input with a fixed tolerance	Numerical agreement under this model and precision, not language quality
Is there less computation?	Count Q/K/V projection rows	These projections decrease, not necessarily every operation
Is the wait shorter?	Time the complete query against a no-cache baseline with repetitions	Local time for this workload, not production LLM performance

Valid prefix hit → fewer token projections ─┐
Signature, lookup, copying, segmented calls ┼→ total query time (measure it)
Cache miss → recomputation ────────────────┘

The three-scope / 16KiB case is a counterexample: longest halves projection rows relative to all, yet remains slower than none. Retain both baselines rather than choosing the slower cache policy to claim acceleration.

Exercise: do 432 rows instead of 864 mean “50% faster”?

No. Only the projection count falls by 50%. The corresponding median is 6.723ms versus 1.697ms without caching. File access, hashing and copying are timed, but have not been profiled separately; none can be declared the dominant bottleneck from these totals.

Change admission, keep the model

The unchanged untrained Decoder has two layers, dimension 16 and vocabulary 32. Each request contains twelve tokens: an eight-token prefix and a changing four-token suffix.

Policy	Persistent checkpoints	Query path
none	None	Compute all twelve tokens; no cache signature or copies
all	Full prefixes at tokens 4 and 8	Find the longest matching prefix and continue
longest	Only the full prefix at token 8	Same lookup, segmented computation and LRU machinery

Both cache policies retain the same segmentation even when token-4 state is not admitted. The no-cache baseline performs genuinely ordinary computation rather than paying artificial lookup costs for symmetry.

Entries own full K/V and position arrays, without shared pages. Persistent payload costs 528 bytes per token: 2,112 bytes for four tokens and 4,224 for eight. This is array payload, not process RSS or GPU memory.

Capacity arithmetic is useful, but not sufficient

For three isolated scopes:

all:      3 × (2,112 + 4,224) = 19,008 bytes
longest:  3 × 4,224           = 12,672 bytes
16 KiB:                       16,384 bytes

The longest checkpoints fit; all checkpoints do not. Identical tokens in different scopes remain isolated.

However, failing to hold every checkpoint does not imply zero hits. At 12KiB, the two-branch all trace evicts one short checkpoint, retains both long prefixes and subsequently hits ten times. Access order and the surviving working set matter alongside capacity.

What the timer includes

Three seeds × three budgets (8, 12, 16KiB) × three traces × three policies produce 81 configurations. Each receives one excluded warm-up trace, then five measured traces. Configuration order is shuffled in each measurement block with a fixed seed; each trace starts with a fresh model object and empty cache.

The archive contains 405 measured traces / 4,860 requests, plus 972 excluded warm-up requests. Each budget/trace/policy summary aggregates fifteen trace times: three model seeds and five repetitions.

start = time.perf_counter_ns()
states, info = store.query(scope, model, tokens)
elapsed_ns = time.perf_counter_ns() - start
fresh, _ = model.full(tokens)  # outside the timing interval

Timing includes signature checks, lookup, copies, computation, admission and eviction. It excludes model construction, reference computation, output serialization and plotting. The inherited signature function reads the model source file and hashes weights on every query, so that source-file read is included. This is intentionally the existing teaching implementation, not a tuned production cache.

perf_counter_ns() provides an integer nanosecond representation for time differences; its unit does not guarantee nanosecond measurement accuracy. Python time reference

At 16KiB, less work does not mean less time

Each row covers a twelve-request trace, including the initial cold request. Times are median and minimum–maximum across fifteen measured traces.

Trace	Policy	Hits	Q/K/V rows	Query time (ms)
Hot prefix	none	0	864	1.646 (1.607–3.260)
Hot prefix	all	11	336	5.921 (5.680–6.886)
Hot prefix	longest	11	336	6.231 (5.679–8.648)
Alternating branches	none	0	864	1.733 (1.603–3.511)
Alternating branches	all	10	384	6.495 (6.036–9.840)
Alternating branches	longest	10	384	6.282 (5.846–7.138)
Three scopes	none	0	864	1.697 (1.612–2.126)
Three scopes	all	0	864	10.263 (9.255–15.599)
Three scopes	longest	9	432	6.723 (6.345–10.020)

Three scopes provide the clearest admission contrast: twenty evictions become zero, rows drop from 864 to 432, and median time falls from 10.263ms to 6.723ms. Yet no caching takes 1.697ms. Improving one cache strategy is not the same as beating the no-cache baseline.

The hot trace also rejects a universal claim that longest-only is faster. Both policies hit eleven times; longest-only has a slightly higher median and overlapping ranges. No statistical significance test was performed, and no fastest repetition was selected for presentation.

An explanation, not a component-level profile

The model's matrices are tiny. The cache path still performs source/weight hashing, Python container management, owned copies and segmented calls. Fresh computation avoids that machinery.

This explanation is consistent with the code and observations, but this study does not time individual components. It cannot assign a percentage of overhead to hashing or copying. A next experiment could profile those parts and compare a model-load-time version signature with per-query signatures while retaining invalidation tests.

Simply deleting identity checks would make stale K/V reusable after weight or positional-rule changes. Returning a wrong answer faster is not an optimization.

Keep the zero-benefit cases

At 8KiB, two longest prefixes require 8,448 bytes, exceeding 8,192. Longest-only reduces evictions from 22 to 11 and explicit copying from 76,032 to 50,688 bytes, but still gets zero hits and saves no projection work.

At 12KiB, three longest prefixes require 12,672 bytes, exceeding 12,288. This trace also remains at zero hits. Admission rules cannot erase the capacity constraint.

These deterministic traces are not sampled user traffic. Their hit counts explain mechanisms rather than predict online hit rates.

Entry points and reproduction

Start with capacity and mechanisms, then equivalence, invalidation, longest-prefix reuse and byte budgets. These are completed offline numerical experiments.

The separate context branch has 24 real Agnes 3.0 calls. The later 128-case similar-distractor protocol is offline preparation only, with no new model results. Do not combine their denominators with these numerical experiments.

In a new directory, prepare Python 3.12 (PowerShell):

git clone https://github.com/VirtualSelect/hohoo-ai-lab.git hohoo-ai-lab-cache
cd hohoo-ai-lab-cache
git checkout 503f5271ef0d5baa12353d6eeda672dec5fc240e
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install numpy==2.2.6
cd experiments/09-cache-admission
..\..\.venv\Scripts\python.exe -m unittest discover -s . -p test_cache.py
..\..\.venv\Scripts\python.exe run.py --out evidence/my-run
..\..\.venv\Scripts\python.exe audit.py evidence/my-run

Expect per-query records and arrays. Check numerical error, projection counts and budget invariants before interpreting time. Use a new output directory. Timing need not exactly match the archive; an error beyond the protocol tolerance is a correctness failure. For missing NumPy, check that installation and execution use the same interpreter.

Audit scope: the current script checks result-file hashes and independently recomputes some metrics, but does not traverse manifest.sources to verify source hashes. Pin the revision as well; an audit pass does not establish source identity, environment identity or production performance.

Appendix: numerical checks and archived files

All 4,860 measured requests are compared with a full reference calculation. Maximum absolute error is 4.16 × 10⁻¹⁶. The first measured repetition of each configuration archives reference and reused outputs: 1,944 arrays in total. Later repetitions retain errors and timing without duplicating arrays.

Six contract tests cover invalid policy, a one-byte capacity boundary, scope separation, model identity changes, return/cache memory isolation and longest-only admission. The independent audit replays token/scope LRU behavior without calling the Store under test.

Use the repository's Python/NumPy environment. Archived versions are Python 3.12.14 and NumPy 2.2.6. The manifest identifies frozen source, runtime, NumPy configuration, timer resolution and hashes. Raw results retain ordering, elapsed nanoseconds, work counters, copying and eviction events.

This was not a dedicated benchmark machine: no CPU affinity, GPU or production inference engine. Timing will vary. The useful discipline is to report correctness, capacity, work and elapsed time together, so the reader can see exactly what improved.

Fix follow-up · 2026-10-07

2026-10-07 correction: the audit now verifies manifest.sources and rejects changed code, missing files and paths outside the repository. The original 4,860 queries and timings remain unchanged; source-integrity checks do not provide new cache-performance results.

Fixed code and regression commands.