LAB / 015
L2–L4 / Attention and cache mechanics
From a two-row hand calculation to a two-layer numeric experiment: causal masks, softmax and future-information leakage.
EXPERIMENT RESULT
01 / QUESTION
From a two-row hand calculation to a two-layer numeric experiment: causal masks, softmax and future-information leakage.
METHOD
An untrained two-layer network, three fixed seeds, full/incremental/chunk/window comparisons.
RESULT
Correct caches stay below 1e-12 error; wrong positions and chunk masks produce detectable differences.
OBSERVATIONS
Prefix/cache projected rows: 1800/144; full/window K/V payload: 12288/2048 bytes.
CONCLUSION
Caches retain transformed states. Slots are not absolute positions, and local caches are not cropped-input recomputation.
LIMITATIONS
No training, language task, online call or hardware-speed benchmark; this network does not represent Agnes.
REPRODUCE
See the pinned repository for code, frozen parameters, raw results and independent auditing.
KEEP EXPLORING
Related content
- Mechanisms
LLM mechanics I: should changing future tokens affect earlier outputs?
From a two-row hand calculation to a two-layer numeric experiment: causal masks, softmax and future-information leakage.
- Mechanisms
LLM mechanics II: what does KV caching save, and how do we verify it?
Compare prefix recomputation, token decoding and chunk caching with output errors, projected-row counts and actual array payloads.
- Mechanisms
LLM mechanics III: why is a four-slot cache not four-token recomputation?
Separate absolute positions, cache slots and retained representations with a two-layer sliding-attention experiment.
- PROJECT / 002
Java LLM Practice
Reproducible Java conversation, retry and persistence boundaries, alongside attention, cache-cost and quantization-error experiments.