The transformer archiecture in language models has popularized the **self-attention mechanism**. This mechanism consists of three key components: 

1. **Key**: A label for the previous tokens, describes what they contain
2. **Value**: The actual information or context that is stored
3. **Query**: The question being asked about the preceding text, this information is not cached

For example, let's say that the preceding context consists of the phrase "My name is Bob", tokenized by word. 

$$
\begin{array}{|c|c|c|}
\hline
\textbf{Order} & \textbf{Tokens} & \textbf{KV Cache} \\
\hline
\text{1} & \text{My} & \boxed{My}\; \\
\hline
\text{2} & \text{My name} & \boxed{My}\;\boxed{name} \\
\hline
\text{3} & \text{My name is} &  \boxed{My}\;\boxed{name}\;\boxed{is} \\
\hline
\text{4} & \text{My name is Bob} & \boxed{My}\;\boxed{name}\;\boxed{is}\;\boxed{Bob} \\
\hline
\end{array}
$$

Each token attends strictly to itself and its preceding tokens. This is enabled by an **attention mask**, which determines which tokens are attended to. 

Large language model inference consists of two stages: the **prefill** and the **decode** stages. The prefill stage processes the context and creates the initial KV-cache, while the decode stage autoregressively generates the output token based on the previous context. During the prefill stage where the KV-Cache is generated, the attention weight for each token is generated by attending to all previous tokens. This would be computationally expensive if we had to recompute from scratch each attention weight. Instead, we can use the KV-Cache to speed up the process. The intermediary key and value cache can be stored and later used to calculate these attention weights. By storing the key and value for previous tokens, we no longer have to restart the attention weight calculation for every new token.

![self-attention](https://files.peter.bio/uploads/2026/05/4cc5349a-15f8-475e-bb5b-b82f8cf0e5ac.png)

Even with these optimizations, inference times are still bottlenecked by the prefill phase. There are different methods for speeding up inference, one obvious optimization is not to attend to all previous tokens, only the ones that are most relavent to the current token (caculated through an **attention score**), this is called **sparse attention**. 

\[
\text{Scaled Attention Score} = \frac{Q \times K^T}{\sqrt{d_k}}
\]

A simple but effective method of sparse attention we will investigate is called **block attention**. In block attention, we chunk the context into separate chunks (these could be separate documents or turns in a chat window). Within each chunk, each token attends to only itself and other tokens.

![block-attention](https://files.peter.bio/uploads/2026/05/7ae0561f-1a3f-43f5-abed-be75d3bf4ce2.png)

Block attention be immensely useful in conjunction with augmented generated retrieval. However, block attention does come with some deficits. One of which being that distant relationships across block boundaries are degraded. To fix this post-processing steps are often required. 

KV-Cache Reuse in Large Language Models

Hello world.

I study the mathematical foundations of machine learning - how models learn, why they generalize, and what makes that learning efficient. My work sits at the intersection of three subfields that I believe are more deeply connected than they are usually treated: mechanistic interpretability, learning theory, and machine learning systems.

The questions I find most compelling are ones that reach across these research areas. What internal computations does a model develop over the course of training, and why those rather than others? Which properties of data, architecture, and optimization determine what is learned, how efficiently, and how reliably it transfers? How do the representations that interpretability research uncovers relate to the generalization behavior that learning theory tries to bound? And how do these abstractions hold up when confronted with the constraints of real systems - memory, throughput, latency, and scale? My intuition is that progress on any one of these questions is bottlenecked by the others and that the most informative answers come from working on them simultaneously.

I am incredibly fond of both theory and experimentation. Too many times theoretical ideas yield no concrete practical applications, and many experiments yield unexplainable results. To frame it as a machine learning problem: theory without applications is of bad directional research taste, while unexplainable empirical result explanations are an irrepeatable process.

This blog is meant to be a working notebook. Some posts will be sketches of arguments I am still trying to formalize. Others will be experimental notes - plots, ablations, anomalies - that I have not yet been able to fully explain. Occasionally there will be more developed pieces that bring theory and experiment together on a single question. None of these are intended as substitutes for full papers. The goal is to make my entire research process visible.

I expect some of my ideas to be wrong. Early feedback, disagreement, and refinement are how informal claims become reliable ones, and I would rather surface ideas while they can still be cheaply revised. Posts will aim for technical soundness and clarity even when they are short and for honesty about what is established versus what is still conjectural.

These notes are written for other researchers, though I hope they remain accessible to anyone curious about why machine learning systems behave the way they do. If something here is useful, wrong, or incomplete, I would be glad to hear about it. Progress on these questions is not an individual undertaking; the field moves forward when its working drafts are shared, not only its finished ones.

Thanks for reading.

Blog

2026

KV-Cache Reuse in Large Language Models

Hello World