Inside China’s Machine · Tokenomics, Episode 7 of 10
The Answer Price = price per token × tokens per attempt ÷ success rate
In long-context agent work, the memory a server stores and re-reads for every token sets serving cost more than model size does, and the public record cannot yet show whether the smaller caches China’s labs have built reach the price list.
On 10 September 2026, DeepSeek released V4.1-Flash, a replacement for its cheaper model. The Chinese release gave memory a section of its own: “新一代模型大幅减少了 KV Cache 缓存的大小，与上一代模型相比，对 HBM 的需求减少到 1/4，对 SSD 的需求减少到 1/8。在 Agent 使用场景中，缓存命中的费用往往占比较高，对 KV Cache 的压缩大幅降低了 Agent 类任务的使用成本。” In our translation: the new model substantially reduces the size of the KV cache; compared with its predecessor, demand for HBM falls to one quarter and demand for SSD to one eighth; in agent use, cache-hit charges often make up a large share of the cost, and compressing the KV cache substantially lowers the cost of agent tasks.
A lab selling a model told its customers that the thing it had made cheaper was the stored memory of the conversation.
Here is our view. In long-context agent work, the cost of a token depends less on how many weights a model has than on the state the server must hold and re-read for every token it writes: its look-back over the transcript. Labs in China and elsewhere have shrunk that state, and their own reports show that shrinking it lowers what long-context work costs to serve. DeepSeek, Kimi, MiniMax and Qwen have published numbers on it. Whether this gives China’s labs a lasting edge on price cannot be known from public accounts yet. The reason is specific: the saving and the price list never move alone. Architecture changes, retraining and tariff changes arrive in the same week, and no source reviewed here gives the cost figure separating them.
The look-back, measured in bytes
Start with what a language model does when it writes. It produces one token at a time. A token is the unit a model reads and writes, roughly a word fragment; how much text one token covers depends on the model family’s tokenizer, so a token in one family is not the same amount of text as a token in another. To choose each new token, the model runs a step called attention: its layers compare the new position with earlier positions in the context and decide what to weight. Doing that from scratch each time would mean recomputing the whole transcript for every new token. Servers avoid this by keeping, for each earlier token, a small record of what attention needs from it. That record is the KV cache. The name comes from the “keys” attention compares against and the “values” it combines. The cache grows with every token of context, and while the model is using it, it lives in the GPU’s high-bandwidth memory, HBM, the scarcest and most expensive memory in the machine.
The weights are the model’s learned numbers. They take a fixed amount of memory per server: loaded once and shared by every user on the machine. The cache is different. It belongs to one conversation, it grows as the conversation grows, and at long contexts, or across many users, it can exceed the room the weights take. So the question for a long-running agent is how many bytes each model stores per token of context, more than how big the model is.
Those bytes can be calculated from each model’s published configuration. Figure 1 does this at one common precision, BF16, which spends two bytes per stored number, for a single sequence. The result is logical stored state. It is not what a server allocates at runtime, not a bill, and not a measure of quality.
The figure shows that model size and memory per token barely track each other. Llama-3.1-70B and Qwen3-32B store state the plain way: every layer keeps keys and values for every token, with eight shared “KV heads”, sets of keys and values that several attention heads can use, to save some room. That is how they reach 40 and 32 gibibytes (a gibibyte is about 1.07 billion bytes) at 131,072 tokens of context, for one user. The models below them on the chart use one of three tricks to keep less.
The first is the latent. DeepSeek’s multi-head latent attention, or MLA, does not store full keys and values. It stores a compressed summary per token, a “latent” vector of 512 numbers plus 64 for position, and reconstructs what attention needs when that state is used. That is how a model as large as DeepSeek-V3 lands at 70,272 bytes per token.
The second is the window. Some layers look back only over the last stretch of tokens and forget the rest. Once the window is full, they hold a fixed amount of state however long the conversation runs. Gemma 3 uses “5 local layers for every global layer, starting with a local layer as the first layer of the model,” with a local window of 1,024 tokens. gpt-oss alternates full layers with windowed ones; in the model card’s words, “the bandwidth is 128 tokens.”
The third is linear attention. Instead of a record per token, a layer keeps a running summary of fixed size and updates it as each token passes, the way a running total replaces a list of receipts. Qwen3-Next does this in 36 of its 48 layers, which is why its per-token growth is the smallest on the chart. Those layers still need a fixed block of recurrent state, even for a short conversation; the saving grows with the transcript.
Two readings follow, and the second is the one most readers miss. First, the design choice moves the memory requirement by more than an order of magnitude at the same context length, which no choice of model size does on this chart. Second, the leaders are not all Chinese. At 131,072 tokens, OpenAI’s gpt-oss-120b stores 4.504 GiB against DeepSeek-V3’s 8.578 under these assumptions, and Qwen3-Next stores less than both. The comparators are already optimised designs, not a naive baseline that China beats by default. The design space is shared.
What shrinking it buys, in the labs’ own numbers
Fewer bytes per token matter because HBM limits both how many conversations a GPU can serve at once and how quickly it can read back what it holds. A smaller cache means more users per GPU, longer contexts per user, or both. The evidence that this turns into throughput comes almost entirely from the labs themselves. Each result is a provider-authored report, not an independent audit, and each has its own boundary.
DeepSeek’s V2 paper of May 2024 is where MLA first appeared, and it reported the effect on the whole system: the design cut the “KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times” against DeepSeek’s own earlier 67B model, on eight H800 GPUs. The baseline was its predecessor, not a matched model using the grouped-query approach Llama uses, and the deployed cache was quantised to an average of six bits. So this is a measure of what changed between two of DeepSeek’s own systems, not of MLA in isolation.
The V3.2 paper of December 2025 went one step closer to money. It ran an actual H800 service comparison of V3.2 against V3.1-Terminus and costed the GPU time at “a rental price of 2 USD per GPU hour.” The long-context cost curves for prefill (reading the prompt) and decode (writing the answer) separate substantially as the token position climbs toward 128K. This is modeled rental cost on an assumed rate, not an invoice and not an all-in cost.
By the V4 paper of 2026, DeepSeek reported that at a 1M-token context V4-Pro uses 10% of V3.2’s KV cache and 27% of its per-token compute. That comparison depends on precision and deployment, and it is not a price ratio. Then came V4.1-Flash, the model in the opening release. Its paper reports a global cache of 890 bytes per token, against 3,514 for V4-Flash, a ratio of 3.948. On that accounting, a million tokens of context come to about 0.83 GiB of global cache. The paper’s boundary is narrower than Figure 1’s: it uses mixed precision and excludes the transient sliding-window state. On DeepSeek’s own series, it is the smallest yet.
The other Chinese labs report the same direction at different boundaries. Kimi’s October 2025 Kimi Linear paper reported “reducing KV cache usage by up to 75% and achieving up to 6×” speed against a matched MLA research baseline; the paper’s own Section 6 calls its up-to-6.3x speed figure theoretical, and it is not a number for the deployed K3. Kimi K3, released in July 2026, carried the idea into a flagship: it “以 3:1 比例混合 KDA 与 Gated MLA” (ours: mixes KDA and Gated MLA in a 3:1 ratio), so roughly three linear-attention layers for each latent-attention layer. MiniMax’s MSA paper of June 2026 reported 28.4x fewer attention operations at 1M tokens “under the same head configuration,” with prefill 14.2x and decode 7.6x faster in wall-clock time on H800, for an experimental model. Its M3 launch made a separate claim on a separate boundary, and the two should not be added together. Qwen3.8’s August 2026 paper reported “7.6×and 4.9×attention-module speedups for prefill and decode, respectively” against a dense baseline at 1M. Those are module-level results, not the whole service and not a price.
The closest non-DeepSeek serving measurement reviewed here is a report by LMSYS of 19 August 2026, which served DeepSeek V4-Pro on H20 GPUs. Its capacity-oriented profile at 1M context reached 177.48 output tokens per second per GPU, with four requests per data-parallel rank. That is a throughput operating point, not a latency promise. As modeled arithmetic only, an assumed USD 2 per GPU-hour at that rate comes to about USD 3.13 per million output tokens of GPU rent. That figure leaves out the GPUs doing prefill, replay, storage, CPU, networking, power, idle time, staff and margin. It shows that the throughput is large enough to matter. It does not show what anyone pays to serve.
So the first half of the call stands on the labs’ own evidence: the look-back can be made dramatically smaller, and smaller look-backs raise what a GPU can serve at long context. What remains is whether agents feel it, and whether anyone passes it on.