Reading Notes of DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

TL;DR: Upper layers reuse memory built by its lower layers, groups of layers share one 4-bit memory, and short-lived local memory is rebuilt instead of stored.

Reference: https://arxiv.org/abs/2609.19969

1. Summary

Long-running AI agents feed models ever-growing inputs, so cost is dominated by reading the prompt (prefill) and storing each token’s attention memory (the KV cache). DeepSeek-V4.1-Flash, a 552B-parameter model reading text and images, shrinks both: its upper layers reuse memory built by its lower layers, groups of layers share one 4-bit memory, and short-lived local memory is rebuilt instead of stored. (1) GPU memory drops to 890 bytes/token, ≈4× below DeepSeek-V4-Flash; (2) the pretrained model roughly matches the ≈3×-larger V4-Pro; (3) after agent training, it beats all compared models, closed ones included, on two coding-agent benchmarks (DeepSWE, Terminal-Bench 2.1).

2. Motivation

Agents call tools repeatedly, so every turn re-sends a growing context; prefill compute and KV-cache storage dominate cost. Earlier work shrinks the cache along a single axis: smaller entries (fewer key/value heads), fewer entries (compressing several tokens into one, as in DeepSeek-V4), or sharing across layers (YOCO, index reuse). Each still leaves per-layer copies, full-depth prefill, or local caches stored for days yet used for minutes. V4.1 combines all three axes and adds fixes for prefill depth and local caches. The cost motivation is clear; that compression preserves quality is asserted rather than measured one component at a time.

---

3. Math

Notation

Symbol	Meaning	Value in the paper
$N$	tokens in the context	up to $10^6$
$t, u, j$	token positions; $j$ also indexes cache entries
$L$	Transformer layers	$40$ ($20$ encoder + $20$ decoder)
$d$	hidden width	$5120$
$w$	sliding-window size (paper: $n_{\text{win}}$)	$128$
$m$	compression ratio: tokens merged per cache entry	$2$ (encoder), $1$ (decoder)
$K$	global cache entries each query reads	$512$
$P$	candidate-pool size	$16{,}384$
$H_{\text{enc}}$	hidden states leaving the last encoder layer (paper: $H_{L/2}$)
$r$	residual streams in mHC (paper: $n$)	$4$
$b$	reasoning-effort level	$1$–$100$
$\kappa(b)$	length-penalty coefficient (paper: $k(b)$)

3a. Prerequisites

Token, layer, hidden state. Text and images are chopped into tokens (word pieces, image patches). A Transformer is a stack of $L$ layers; each layer updates one vector per token, the hidden state $h_t\in\mathbb{R}^d$. A layer has two parts: attention (tokens read each other) and a feed-forward network (each token processes its own vector).

Attention. Intuition: each token poses a question (query $q_t$), compares it with a label on every earlier token (key $k_j$), and takes a weighted average of their contents (values $v_j$); better matches get more weight.

$$o_t=\sum_{j\le t}\alpha_{t,j}\,v_j,\qquad \alpha_{t,j}=\frac{\exp(q_t\cdot k_j)}{\sum_{j'\le t}\exp(q_t\cdot k_{j'})}$$

(the usual $1/\sqrt{\text{dim}}$ scaling is folded into $q$). Example: $q_t=1$, $k_1=0$, $k_2=\ln 3$, $v_1=10$, $v_2=20$. The weights are proportional to $(1,3)$, i.e. $(0.25,0.75)$, so $o_t=2.5+15=17.5$.

KV cache. Intuition: past keys and values never change, so the model stores them once and reuses them for every later token. Its size is

$$\text{cache bytes}=N\times\underbrace{\textstyle\sum_{\text{layers that store}}\text{bytes per token}}_{\text{bytes/token}} .$$

Example: 40 layers each storing 1,024 numbers per token at 1 byte each give 40,960 bytes/token; at $N=10^6$ that is ≈41 GB for one conversation.

Prefill vs. decode. Prefill pushes a new prompt through the model in parallel and writes its cache; its cost is proportional to (new tokens) × (layers they pass). Decode generates one token at a time, and each step reads the cache. Agent workloads are input-heavy: every tool call returns long text, so prefill dominates.

Memory tiers. HBM (high-bandwidth memory on the GPU) is fast, small and expensive; the runtime cache of active requests lives there. Host DRAM (CPU memory) and SSD (flash disk) are larger and slower; they hold the persistent cache, kept between turns so a returning conversation skips re-prefilling its prefix (the already-seen beginning). Finding a stored prefix is a cache hit.

Mixture of Experts (MoE). Each feed-forward layer holds many small “expert” networks, and a router sends each token to a few. Total parameters measure capacity; activated parameters measure compute per token. Example: V4.1-Flash has 1 shared + 384 routed experts and picks 6 routed ones, so each token uses 7 of 385 experts (≈1.8%).

Sliding-window attention (SWA). Each token attends only to its last $w$ tokens, so each layer’s SWA cache never exceeds $w$ entries. Information still travels further by hopping up to $w$ tokens per layer, so after $L$ stacked layers the receptive field (how far back a token can be influenced) is

$$\text{receptive field}=L\,w .$$

Example: $40\times128=5{,}120$ tokens.

Global branch = compression + sparse selection (inherited from DeepSeek-V4). To see the whole context cheaply, the model (i) merges every $m$ consecutive tokens into one main-KV entry; (ii) runs a tiny indexer with its own small indexer query $q^{I}$ and indexer key $k^{I}$ to score all entries; (iii) keeps the $K$ best entries (top-K) and attends to them plus the SWA window. Global KV means main KV + indexer K; it grows with $N$ and dominates HBM for long contexts.

Low-precision numbers. BF16, FP8 and FP4 store a number in 16, 8 and 4 bits. Quantization stores a small code plus a shared scale, $x\approx s\cdot c$; component C4 gives the details.

Training vocabulary. Pre-training: predict the next token on a huge corpus, producing a base model. Post-training: SFT (supervised fine-tuning on demonstrations), RL (reinforcement learning: try, receive a reward, reinforce high-reward behavior), OPD (on-policy distillation: the student writes, teacher models score every token). Scaffold or harness: the program around the model that supplies tools (shell, file editor), returns tool outputs and manages context (Claude Code, Codex, DeepSeek Harness). Rollout: one attempt at a task.

3b. Problem setup

Setting 1 — cheap long-context serving.

• Input: multi-turn agent sessions; at each turn a prompt $x_{1:N}$ (text + images) whose first $N_{\text{hit}}$ tokens were cached earlier, plus a new suffix. Output: generated tokens.
• Task: keep quality while minimizing four costs: prefill compute for uncached tokens, decode compute per output token, HBM bytes/token of global KV, and SSD/host bytes/token of persistent KV.
• Reference: DeepSeek-V4-Flash (284B total, 13B activated; every compressed-attention layer keeps its own global KV; FP8 main KV; SWA states saved in the persistent cache).
• Out of scope: measured throughput, latency, price and energy; the paper reports byte counts and weighted FLOP counts.

Setting 2 — capability.

• Base model: few-shot prompts (a handful of solved examples, then the question) on knowledge, reasoning, code/math, long-context and multimodal benchmarks, plus text compression on internal held-out corpora. References: V4-Flash-Base and V4-Pro-Base (1.6T).
• Post-trained model: reasoning tests and agentic tasks run in sandboxed Linux containers with a harness, at maximum reasoning effort. References: Opus-5 and GPT-5.6 Sol (closed); Kimi-K3, GLM-5.3, V4-Pro, V4-Flash (open).

Setting 3 — controllability and deployment.

• Effort: the system prompt carries “Reasoning Effort: $b$” with $b\in\lbrace 1,\dots,100\rbrace$; the study measures accuracy against output length.
• Scaffold transfer: the same checkpoint inside 8 harness configurations.
• Multi-agent: a lead agent plus teammates under a wall-clock deadline, compared with a single agent.

3c. Evaluation metrics

Global KV bytes per token, $B_{\text{glob}}$ (lower is better). Intuition: how much GPU memory each context token occupies while its request runs; it caps how many long requests fit on a GPU. Only layers that produce a cache count, each divided by its compression ratio:

$$B_{\text{glob}}=\sum_{f\in\mathcal{F}}\frac{\beta_{\text{main}}+\beta_{\text{idx}}}{m_f},$$

with $\mathcal{F}$ the cache-producing layers and $\beta$ the bytes per entry. Example: two producing layers with 100-byte entries, one at $m=2$ and one at $m=1$, give $50+100=150$ bytes/token. (The real computation, giving 890, is in C2.)

Persistent bytes per token (lower is better). The same count for SSD/host memory between turns; it caps how many conversations stay resumable.

Decode FLOPs per token, precision-weighted (lower is better). Intuition: arithmetic per generated token, with cheaper low-precision operations discounted.

$$F=F_{\text{BF16}}+0.5\,F_{\text{FP8}}+0.25\,F_{\text{FP4}} .$$

Example: 10 G BF16 + 20 G FP8 + 8 G FP4 operations give $10+10+2=22$ GFLOPs.

Pass@1 / Resolved (higher is better). Intuition: the chance a single attempt succeeds, averaged over tasks $\mathcal{T}$ and $S$ samples per task.

$$\text{Pass@1}=\frac{1}{|\mathcal{T}|}\sum_{x\in\mathcal{T}}\frac{1}{S}\sum_{s=1}^{S}\mathbf{1}\big[\text{attempt } s \text{ on } x \text{ succeeds}\big].$$

Example: 2 tasks, 3 samples; task A succeeds 2 of 3 times, task B 1 of 3 → Pass@1 $=0.5$. Resolved (DeepSWE) is Pass@1 where success means the hidden tests pass. Pass@5 counts a task as solved if any of 5 attempts succeeds.

Exact match (EM) and F1 (higher is better). EM is the fraction of answers identical to the reference. F1 gives partial credit for word overlap:

$$F1=\frac{2PR}{P+R},\quad P=\frac{\text{shared words}}{\text{predicted words}},\quad R=\frac{\text{shared words}}{\text{reference words}} .$$

Example: prediction “the red car”, reference “red car” → $P=2/3$, $R=1$, $F1=0.8$.

Bits per byte, BPB (lower is better). Intuition: how many bits the model needs to encode raw text; a better model of the text is less surprised by it.

$$\text{BPB}=\frac{-\sum_{t}\log_2 p(x_t\mid x_{<t})}{\text{number of bytes of text}} .$$

Example: 4 tokens, each predicted with probability $0.5$, cost $4$ bits; spread over 8 bytes, BPB $=0.5$.

Codeforces rating (higher is better). An Elo-style score from competitive-programming contests; above 3,000 is the top human tier.

Acc@0.5 (visual grounding, higher is better). A predicted box counts as correct if its overlap with the true box reaches one half:

$$\text{IoU}=\frac{\text{area}(A\cap B)}{\text{area}(A\cup B)}\ \ge 0.5 .$$

Example: two 10×10 boxes overlapping in a 10×5 strip have $\text{IoU}=50/150=0.33$ → incorrect.

Almost@1 and Mean@5 (multi-agent, higher is better). Almost@1 is the fraction of single rollouts scoring at least $0.95$ (a nearly complete program rebuild); Mean@5 is the average score over 5 rollouts. Example: rollout scores $0.97, 0.80, 0.96$ → Almost@1 $=2/3$.

Output tokens (lower is cheaper). Mean tokens generated per response or trajectory; the cost side of the effort trade-off.

3d. The standard approach and why it fails

The standard design is dense attention: every layer stores full keys and values for every token, and prefill runs every token through every layer.

$$B_{\text{dense}}=L\times(\text{key+value numbers per layer})\times(\text{bytes per number}),\qquad \text{Prefill}\propto N_{\text{new}}\cdot L .$$

Example: DeepSeek-V1 (2023) used 389,120 bytes/token, so a 1M-token context needs ≈389 GB of HBM for one request.

Five failures follow, and every component in 3f targets at least one:

• F1 — per-layer copies: each layer keeps its own cache, so bytes grow with $L$.
• F2 — full-depth prefill: every uncached token passes all $L$ layers.
• F3 — scanning cost: per-query attention or indexing work grows with $N$.
• F4 — mismatched local state: SWA states are needed for minutes yet stored for days, and rebuilding them exactly needs $L\,w$ tokens.
• F5 — wide numbers: 16 or 8 bits per cached number.

DeepSeek-V4-Flash, the reference, already shrinks entries and merges tokens (partly fixing F3 and F5) to reach 3,514 bytes/token, but keeps per-layer copies (F1), full-depth prefill (F2) and persistent SWA states (F4).

3e. Existing baselines

Method	Axis it compresses	What it fixes	Limitation
GQA / MLA (DeepSeek-V2, V3)	entry size: fewer KV heads / one small latent shared by heads	smaller entries	still one cache per layer (F1)
CSA / HCA (DeepSeek-V4)	sequence: $m$ tokens → 1 entry, then top-$K$	F3	each layer owns its cache (F1); needs a hybrid of two attention types
Cross-layer attention	layers: neighbors share KV	F1 for storage	fixed pairing, no sparse selection
YOCO	layers: the upper half reads one KV made by the lower half	F2 (halves prefill)	one cache for all upper layers limits capacity; upper layers’ local states lose depth
IndexCache / YOIO	layers: reuse top-$K$ indices	indexer compute	no storage saved; one network-wide selection hurts quality
HySparse	layers: sparse layers reuse dense layers’ KV	F1 partly	keeps full-attention layers
Zero SWA Caching (V4 report)	deployment: drop SWA from storage, rebuild exactly	F4 storage	exact rebuild replays $L\,w$ tokens, too slow in production
MTP (DeepSeek-V3)	decoding: draft tokens verified in parallel	decode speed	drafter trained jointly through all of pre-training

None of these covers all three compression axes (entry size, sequence, layers) at once.

3f. The method: DeepSeek-V4.1-Flash

One-sentence idea. Make each context token leave behind as little memory as possible: build the upper half’s global memory from the lower half’s output (CED), let groups of layers share one memory and one selection (CSA2), store that memory in 4 bits (FP4), and stop archiving short-lived local states, rebuilding them approximately from only the last $w$ tokens (Bounded Replay).

Analogy. A 40-floor library. In the old building every floor kept its own photocopy of the card catalog. Now only four floors keep a catalog, written in shorthand; the other floors borrow the nearest one, and many also borrow its list of relevant cards. Desk scratch-notes (local states) are no longer archived in the basement; when a reader returns, each floor rewrites them from the last page read.

Layer map (zero-indexed; Full, Reindex and Reuse are defined in C2). Every layer also has its own SWA ($w=128$) and an MoE feed-forward.

Layers	Part	Global attention	$m$
0–1	encoder	none (SWA only)	—
2–7, 8–13, 14–19	encoder, 3 groups	Full + 5 × Reuse	2
20–23	decoder, group 1	Full (built from $H_{\text{enc}}$) + 3 × Reuse	1
24–27, 28–31, 32–35, 36–39	decoder, 4 groups	Reindex + 3 × Reuse	1

C0. Vision input. A vision encoder (DeepSeek-ViT: 32 layers, width 1024, 14-pixel patches) turns an image into a grid of patch features; a 3×3 pixel-unshuffle folds each 3×3 neighborhood into one vector, and a 2-layer MLP maps it to width $d$. Intuition: nine neighboring patches become one token, so images cost 9× fewer tokens.

$$\text{image tokens}=\frac{(H_{\text{px}}/14)\,(W_{\text{px}}/14)}{9}$$

Example: a 1344×1344 image has $96\times96=9{,}216$ patches → $1{,}024$ tokens. Image and text tokens then flow through the same backbone.

C1. Causal Encoder–Decoder (CED) → fixes F2. Intuition: the decoder’s global memory does not need the decoder’s own computation; a learned projection of the encoder’s final output suffices. Uncached tokens then need only the 20 encoder layers during prefill.

$$c_t=h^{\text{enc}}_t\,W^{KV},\qquad g_t=h^{\text{enc}}_t\,W^{Z},$$

where $c_t$ is the raw main-KV vector and $g_t$ its compression weights (C2). Only the decoder’s single Full layer performs this projection; the other decoder layers borrow its result. Each decoder layer still computes its own SWA keys and values from its own hidden state, so local processing keeps full depth. Prefill cost becomes

$$\text{Prefill}_{\text{CED}}\propto N_{\text{new}}\cdot\tfrac{L}{2}+w\cdot\tfrac{L}{2}\qquad\text{instead of}\qquad N_{\text{new}}\cdot L,$$

where the $w\cdot L/2$ term is the decoder replay of C5. Example: $N_{\text{new}}=20{,}000$, $L=40$, $w=128$ gives $400{,}000+2{,}560=402{,}560$ token-layers instead of $800{,}000$, i.e. 50.3%. This is why the model activates ≈8B parameters per token in prefill but 16B in decode.

C2. Compressed Sparse Attention 2 (CSA2) → fixes F1 and F3.

Step 1 — compress. Intuition: merge each block of $m$ tokens into one entry by a learned weighted average.

$$\bar c_j=\sum_{u\in\mathcal{B}_j}\operatorname{softmax}_{u\in\mathcal{B}_j}(g_u)\odot c_u,\qquad \mathcal{B}_j=\lbrace (j-1)m+1,\dots,jm\rbrace .$$

CSA2 uses non-overlapping blocks and no absolute position embedding (V4’s CSA pooled $2m$ overlapping tokens). Example ($m=2$, one weight per token for simplicity): $c_1=(1,4)$, $c_2=(3,0)$ with weights $(0.75,0.25)$ give $\bar c_1=(1.5,3)$. At $m=1$ (decoder) entries stay uncompressed.

Step 2 — index. Intuition: a cheap 128-dimensional score decides which entries deserve full attention. CSA2 derives the indexer key from the entry itself, $k^{I}_j=\bar c_j W^{I}$, and scores with the lightning-indexer form of DeepSeek-V3.2/V4:

$$\sigma_t(j)=\sum_{h=1}^{32}\omega_{t,h}\,\operatorname{ReLU}\big(q^{I}_{t,h}\cdot k^{I}_j\big),\qquad S_t=\operatorname{TopK}_{j}\ \sigma_t(j),\quad |S_t|=K=512 .$$

Example (one head, $\omega=1$): $q^{I}=(1,-1)$ and keys $(2,0),(0,1),(1,1)$ score $2,0,0$, so top-1 picks entry 1.

Step 3 — attend. Intuition: each layer’s own query reads the selected global entries plus its own local window.

$$o^{(l)}_t=\operatorname{Attn}\Big(q^{(l)}_t;\ \lbrace\bar c_j\rbrace_{j\in S_t}\ \cup\ \lbrace(k^{(l)}_u,v^{(l)}_u)\rbrace_{t-w<u\le t}\Big).$$

Example: at $t=10^6$ the query reads $512+128=640$ entries instead of $10^6$.

Step 4 — share across layers with three modes. Intuition: neighboring layers want nearly the same memory and often the same selection, so only one layer per group pays for them.

Mode	Main KV and indexer K	Indexer query and scoring	Selection $S_t$
Full	computes its own	yes	new
Reindex	borrows from the latest Full layer	yes (own indexer query)	new
Reuse	borrows from the latest Full layer	no	borrows the latest one

Every mode still computes its own query $q^{(l)}$ and its own SWA cache.

Resulting memory (derived from the configuration; matches the paper’s 890 exactly). Only the 4 Full layers store global KV: 3 encoder layers at $m=2$ and 1 decoder layer at $m=1$, i.e. $3\cdot\tfrac12+1=2.5$ entries per token. A main entry has 512 channels in FP4 with a 1-byte scale per 16 channels: $256+32=288$ bytes. An indexer key has 128 channels in MXFP4 with a 1-byte scale per 32 channels: $64+4=68$ bytes. Hence

$$B_{\text{glob}}=2.5\times(288+68)=890\ \text{bytes/token},$$

against 3,514 for V4-Flash (3.9× less) and 389,120 for V1 (437× less). A 1M-token context then holds ≈0.89 GB of global KV.

C3. Hierarchical Sparse Indexer (decoder only) → fixes F3 for the deeper indexers. Intuition: the first decoder indexer already knows roughly where the relevant material lies, so later indexers search only that neighborhood. The Full layer scores each block $\beta$ of 8 consecutive positions by its best member, keeps the top 2,048 blocks as a candidate pool, and each later Reindex layer selects its own top-$K$ inside the pool:

$$\text{score}(\beta)=\max_{j\in\beta}\sigma_t(j),\qquad \mathcal{P}_t=\bigcup_{\beta\in\operatorname{Top}_{2048}}\beta\ \ \big(|\mathcal{P}_t|\le 16{,}384\big),\qquad S^{(l)}_t=\operatorname{TopK}_{j\in\mathcal{P}_t}\ \sigma^{(l)}_t(j).$$

Example at $N=10^6$: decoder scoring per query falls from $5\times10^6$ entries (5 indexers × full scan) to $10^6+4\times16{,}384\approx1.07\times10^6$, a 4.7× saving; each Reindex layer alone scores 61× fewer entries. The pool restriction is also applied during post-training, so deeper indexers learn under the same search space they meet at inference.

C4. FP4 main KV cache → fixes F5. Intuition: store each cached number as a 4-bit code times a scale shared by 16 channels. Codes come from the E2M1 grid $\lbrace 0,\pm0.5,\pm1,\pm1.5,\pm2,\pm3,\pm4,\pm6\rbrace$; the scale is an 8-bit E4M3 number (maximum 448).

$$s=\frac{\max_{i\in\text{block}}\lvert x_i\rvert}{6},\qquad c_i=\operatorname{round}_{\text{E2M1}}(x_i/s),\qquad \hat x_i=s\,c_i .$$

Example (4 of the 16 values): $x=(2.4,\ 1.1,\ -0.65,\ 0.05)$ gives $s=0.4$ and $x/s=(6,\ 2.75,\ -1.625,\ 0.125)$, which round to codes $(6,3,-1.5,0)$ and decode to $\hat x=(2.4,\ 1.2,\ -0.6,\ 0)$. Storage is $(16\times4+8)/16=4.5$ bits per number instead of 8 for FP8, nearly halving the cache. The largest representable value, $448\times6=2688$, far exceeds the cache’s bound of $\sqrt{512}\approx22.6$ (normalized 512-channel vectors), so no second-level global scale is needed. The model learns to tolerate the rounding through quantization-aware training (QAT: rounding is simulated during post-training). SWA caches stay in FP8 because they are more sensitive to rounding.

C5. SWA Bounded Replay → fixes F4 and completes F2. Intuition: a stack of windowed layers formally depends on $L\,w$ past tokens but in practice on far fewer, so the missing local states are rebuilt from only the last $w$ tokens, with each window cut at the replay start $s$:

$$\text{SWA keys for query } i:\quad \big[\max(s,\ i-w+1),\ i\big].$$

Example: $s=1000$, $w=128$, $i=1050$ → keys $1000..1050$ (51 keys) instead of $923..1050$ (128 keys). Replay cost drops from $L\,w=5{,}120$ tokens to $w=128$ (40× less).

Two uses:

• Encoder replay (storage). SWA states leave the persistent cache and live in a small host-DRAM pool (10% of each machine’s DRAM, minute-scale lifetime); global KV stays on SSD for at least 72 hours. On a global hit with an SWA miss, the encoder replays the last $w$ cached tokens to regenerate SWA states only, reusing the stored global KV, and then processes the new suffix.
• Decoder replay (compute). At every prefill, only the last $w$ prompt tokens run through the 20 decoder layers to create the decoder SWA states that decoding needs, instead of $\tfrac{L}{2}w=2{,}560$ tokens. Post-training simulates this replay so the model adapts to it.

Intuition for the persistent saving: SWA filled about half of V4-Flash’s persistent cache, and the remaining global part shrinks 4×.

$$\frac{P_{4.1}}{P_{4}}\approx\frac{B^{4.1}_{\text{glob}}}{2\,B^{4}_{\text{glob}}}=\frac{890}{2\times3514}\approx\frac{1}{7.9}\approx\frac18 .$$

The price is that rebuilt states, and therefore later outputs, depend slightly on where the cache hit occurred.

C6. Single-Pass mHC → halves activation memory traffic. Intuition: mHC (manifold-constrained hyper-connections) passes $r=4$ parallel residual streams $X_l\in\mathbb{R}^{r\times d}$ (the running notes carried between layers) and learns, per token, how to mix them into the block input ($A_l$), how to remix them ($B_l$) and where to write the block output ($C_l$). In V4 the input mix $A_l$ needs a full pass over $X_l$ before the block can start, forcing a second read. Single-Pass mHC uses the previous block’s mix instead:

$$X_{l+1}=B_lX_l+C_l\,F_l\big(A_{l-1}X_l\big),\qquad (A_l,B_l,C_l)=\mathcal{H}(X_l).$$

Example ($r=2$, $d=2$): $X_l=\begin{pmatrix}1&2\\3&4\end{pmatrix}$ and $A_{l-1}=(0.5,0.5)$ give the block input $(2,3)$ immediately, while $\mathcal{H}$ reads the same values to produce the next block’s coefficients. Activation traffic drops from $(4r+4)d$ (V4’s kernels) to the lower bound $(2r+2)d$: with $r=4$, $d=5120$, from $102{,}400$ to $51{,}200$ numbers moved per token per layer. A fused kernel, Mega-mHC, achieves this; the shift costs negligible quality according to the paper.

C7. Engram → extra memory without extra compute. Intuition: huge lookup tables remember frequent short phrases. The last 2, 3 and 4 tokens are hashed into table rows, and the fetched vectors are added to the hidden state through a learned gate. Lookups cost almost no arithmetic, and their addresses depend only on the input tokens, so rows are prefetched from CPU memory. In simplified form:

$$e_t=\Big[\,E_{n,k}\big[\operatorname{hash}_{n,k}(x_{t-n+1:t})\big]\,\Big]_{n\in\lbrace 2,3,4\rbrace,\ k=1..8},\qquad h_t\leftarrow h_t+\gamma_t\odot(e_tW_E),$$

with $\gamma_t$ a context-dependent gate. Example: the parameter count is $2$ modules $\times\,3$ orders $\times\,8$ heads $\times\,16$M rows $\times\,256$ dims $\approx196.6$B, matching the paper’s 196B. The modules sit at layers 1 and 14 and are stored in FP8.

C8. DSpark speculative decoding → faster decoding. Intuition: a small 3-block drafter guesses the next 5 tokens in one pass; the big model checks several at once; a confidence head estimates each guess’s chance of surviving, and a scheduler picks how many to verify given current server load. With per-position acceptance probabilities $p_i$:

$$\pi_i=\prod_{u\le i}p_u,\qquad \mathbb{E}[\text{tokens per check of length }k]=1+\sum_{i=1}^{k}\pi_i,\qquad k^{\star}=\arg\max_k\frac{1+\sum_{i\le k}\pi_i}{\text{cost}(k)} .$$

Example: $p=(0.9,0.8,0.6,0.3,0.2)$ gives $\pi=(0.9,0.72,0.43,0.13,0.03)$. With a toy $\text{cost}(k)=1+0.1k$, tokens per unit cost are $1.73,\,2.18,\,2.35,\,2.27,\,2.14$ for $k=1..5$, so the scheduler verifies $k^{\star}=3$. (The “+1” is the token the verifier always produces; the paper’s scheduler uses profiled engine-throughput curves in place of the toy cost.) DSpark is trained after pre-training on a frozen backbone, then alongside the backbone in post-training without sending gradients into it.

C9. Training recipe.

Data and schedule. 45T tokens with a text-only : multimodal ratio of 7:1; sparse attention from the first step at 64K context, extended to 1M at 34T tokens; batch 100.6M tokens (≈447K steps). Intuition for the learning rate: warm up over 2,000 steps to $\eta_{\max}=2.6\times10^{-4}$, hold until 28T tokens, glide down a half-cosine to $\eta_{\min}=2.6\times10^{-5}$ by 40T, then hold until 45T.

$$\eta(T)=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\Big(1+\cos\tfrac{\pi(T-28)}{12}\Big),\qquad T\in[28,40]\ \text{(trillions of tokens)}.$$

Example: $T=34$ gives $\cos(\pi/2)=0$, so $\eta=1.43\times10^{-4}$.

Per-modality load balancing. Intuition: image and text tokens prefer different experts, so each modality keeps its own routing bias $\delta^{\text{mod}}_e$ per expert, nudged up for under-used experts and down for over-used ones; biases influence only which experts are chosen, not their output weights.

$$\delta^{\text{mod}}_e\leftarrow\delta^{\text{mod}}_e+0.001\cdot\operatorname{sign}\big(\bar n^{\text{mod}}-n^{\text{mod}}_e\big),$$

with $n_e$ the expert’s token count in the step and $\bar n$ the average. Example: image tokens send 50 tokens to expert 3 against an average of 20, so expert 3’s image bias drops by 0.001.

Optimizers. Intuition for Muon: update a weight matrix with the “orthogonalized” momentum so every direction takes an equal-sized step. For momentum $M=U\Sigma V^{\top}$,

$$\Delta=UV^{\top}.$$

Example: $M=\operatorname{diag}(4,1)$ gives $\Delta=\operatorname{diag}(1,1)$. Head-wise Muon applies this separately to each attention head’s slice of the query and key weights. For the giant table-like matrices (Engram tables, token embedding, output head; $V$ rows by $d$ columns), a Sinkhorn-balanced update alternately rescales the momentum’s rows and columns to unit norm for 11 rounds, multiplies by $\sqrt d$ so rows have root-mean-square 1, and uses learning rate $0.18\,\eta$:

$$\Delta=\sqrt d\,D_r\hat G D_c,\qquad \tfrac1d\textstyle\sum_{j}\Delta_{ij}^2\approx1,\qquad \tfrac1V\sum_{i}\Delta_{ij}^2\approx1 .$$

Example ($d=2$): momentum rows $(3,4)$ for a frequent token and $(0.3,0.4)$ for a rare one both become $(0.6,0.8)\cdot\sqrt2=(0.85,1.13)$, with RMS 1, so rare rows get equally sized updates; only one momentum buffer is stored (Adam would store two).

Post-training. SFT → RL → OPD with standard algorithms; the effort goes into data. The model itself builds tasks as (problem, environment, verifier) triplets filtered for difficulty and correctness; cooperating agents turn internal coding sessions and starred GitHub repositories into verified container tasks; mocked enterprise tools reproduce real workflows and failures; millions of concurrent sandboxes run on DSec, a custom container platform; RL runs across many scaffolds, with model merging between runs; a final OPD distills from 40+ teacher models. Rollouts are generated asynchronously with bounds on staleness.

Reasoning-effort control. Intuition: during RL the reward subtracts a length penalty that is harsh at low effort and gentle at high effort, so the model learns to think longer when asked. For response $j$ at effort $b$ with $\ell_{b,j}$ reasoning tokens:

$$r_{b,j}=r^{\text{task}}_{b,j}-\min\Big(C_{\max},\ \kappa(b)\frac{\ell_{b,j}}{L_{\text{norm}}}\Big),\qquad \kappa(b)=\kappa_0\,e^{-(b-b_{\min})/\tau},$$

where $\tau$ equals a constant times the average spacing between training effort levels. Rewards are compared only within the same prompt and effort, $A_{b,j}=r_{b,j}-\operatorname{mean}_{j'}r_{b,j'}$, so short and long efforts never compete directly. Example with $\kappa_0=0.5$, $b_{\min}=25$, $\tau=25$, $L_{\text{norm}}=10$k, $C_{\max}=1$ and $\ell=20$k tokens: the penalty is $1.0$ at $b=25$, $0.37$ at $b=50$ and $0.05$ at $b=100$.

Why exponential: if extra thinking helps with exponentially shrinking returns, $p'(\ell)=a\,e^{-\ell/s}$, the reward-maximizing length solves $p'(\ell^{\star})=\kappa(b)/L_{\text{norm}}$, giving

$$\ell^{\star}(b)=s\log\frac{a\,L_{\text{norm}}}{\kappa_0}+\frac{s}{\tau}(b-b_{\min}),$$

a straight line in $b$. Example: $s=4$k tokens and $\tau=25$ mean every +25 effort adds ≈4k reasoning tokens. The public API exposes low / high / max as $b=50/75/100$.

Agent Team reward. Intuition: reward teamwork, but charge for time along the slowest chain of dependent steps (the critical path of the execution graph), so parallel work is cheap and waiting is expensive.

$$r=r^{\text{task}}+r^{\text{collab}}-\mu\cdot\max_{\text{paths }\rho}\ \sum_{e\in\rho}\text{time}(e).$$

Example: two teammates work in parallel for 10 and 6 minutes, then the lead tests for 3; the latency is $10+3=13$ minutes, not $19$. Event times come from token counts at fixed prefill/decode rates plus measured tool time.

Pseudocode — serving one agent turn.

SERVE_TURN(prompt x[1..N], ssd_cache, dram_pool):
  h ← longest prefix of x whose global KV is in ssd_cache          # global hit
  if encoder SWA states for x[1..h] are not in dram_pool:
      ENCODER(x[h-w+1..h], window_start = h-w+1, writes = SWA only)  # encoder bounded replay
  ENCODER(x[h+1..N])                  # 20 layers; Full layers 2, 8, 14 write FP4 global KV (m = 2)
  decoder global KV ← project(H_enc[h+1..N])   # layer 20's FP4 global KV (m = 1); no decoder compute
  DECODER(x[N-w+1..N], window_start = N-w+1)   # decoder bounded replay: decoder SWA states only
  store new global KV → ssd_cache (≥ 72 h);  encoder SWA states → dram_pool (minutes)
  repeat until the turn ends:                  # decoding: all 40 layers, 16B activated
      for each layer:
          Full    → score all entries, keep top-512 (decoder: also build the 16,384-position pool)
          Reindex → score the pool only, keep top-512
          Reuse   → borrow the latest selection
          attend to 512 global entries + 128 SWA entries
      DSpark drafts 5 tokens; the model verifies the k* most promising

How each component fixes each failure.

Failure	Fix	Effect
F1 per-layer copies	CSA2 sharing: 4 cache-producing layers out of 40	global KV 3,514 → 890 bytes/token (together with F5)
F2 full-depth prefill	CED + decoder bounded replay	prefill ≈ halved; 8B activated in prefill
F3 scanning cost	top-512 selection, index reuse, hierarchical pool	decode FLOPs rise ≈25% from 4K to 1M context
F4 mismatched local state	SWA removed from SSD; encoder bounded replay	persistent cache ≈1/8 of V4-Flash
F5 wide numbers	FP4 main KV with per-16-channel scales	4.5 instead of 8 bits per number

---

4. Results and analysis

Organization. The experiments answer three questions in sequence. Part A asks whether the architecture is actually cheaper (memory and compute). Part B asks whether that cheapness costs capability, first for the pretrained base model, then for the post-trained agent. Part C asks whether users can steer cost and deploy the model flexibly (reasoning effort, different harnesses, agent teams). General setup: base models are evaluated few-shot in DeepSeek’s internal framework with identical settings; post-trained models run at maximum effort ($b=100$) with temperature 1.0; coding agents run in DeepSeek Harness Minimal mode (a single bash tool) with a 1M-token context and top-p 0.95, except DeepSWE (mini-SWE harness) and SEC-Bench Pro and visual-agent tasks (Claude Code harness, 512K context); AutomationBench and Agents’ Last Exam use their official scaffolds; containers have no internet, no git history and purged package caches to limit reward hacking. How to read the tables: rows are benchmarks or quantities, columns are models or settings; Pass@1, EM, ratings and scores are better when higher; bytes, FLOPs, BPB and output tokens are better when lower; bold marks the best entry in a row; ≈ marks values read off a figure.

Part A — Is it cheaper?

Question → setup. How much memory and compute does each context token cost, compared with earlier DeepSeek generations? The paper counts global-KV bytes per token for each generation (Figure 1b), computes precision-weighted decode FLOPs per token for contexts from 4K to 1M (Figure 2), and derives prefill and persistent-cache savings from the architecture (metrics: $B_{\text{glob}}$, persistent bytes, weighted FLOPs, activated parameters). How to read: lower is better; a flatter FLOPs-vs-context curve means long contexts stay cheap.

Quantity (lower is better)	DeepSeek-V1	V3.2	V4-Flash	V4.1-Flash
Global KV, bytes/token (HBM)	389,120	48,068	3,514	890
Decode GFLOPs/token at 4K context	≈150	≈80	≈18	≈22
Decode GFLOPs/token at 1M context	≈3,000	≈700	≈60	≈27
Persistent KV relative to V4-Flash	—	—	1	≈1/8
Activated parameters, prefill / decode	—	—	13B / 13B	8B / 16B

• Takeaway A1 — memory per token falls ≈4×. Global KV is 890 bytes/token, 3.9× below V4-Flash and 437× below V1; a 1M-token request holds ≈0.89 GB instead of ≈3.5 GB. Removing SWA states from storage brings the persistent cache to ≈1/8 of V4-Flash’s.
• Takeaway A2 — decode cost is nearly flat in context length. From 4K to 1M tokens, decode FLOPs grow ≈25% (≈22 → ≈27 GFLOPs) versus ≈3.3× for V4-Flash. The growth is explained by the four Full-layer indexers alone (derived from the configuration): they scan $2.5N$ entries at ≈$32\times128\times2\times0.25=2{,}048$ weighted FLOPs each, i.e. $2.5\times10^6\times2{,}048\approx5$ GFLOPs at $N=10^6$, matching the ≈5-GFLOP rise. Below ≈100K tokens V4.1 is slightly costlier per token (≈22 vs ≈18 GFLOPs) because decode activates 16B parameters against V4-Flash’s 13B.
• Takeaway A3 — prefill compute is roughly halved. Uncached tokens pass only the 20 encoder layers (≈8B activated), plus a fixed 128-token decoder replay. This is an analytic count; the paper reports no wall-clock throughput or latency.

Part B — Does compression cost capability?

Question → setup. With 4 cache-producing layers and 4-bit memory, does the model still know and do as much? B1 compares base models on few-shot benchmarks and on bits-per-byte over three internal held-out corpora (internal documents, internal code repositories, academic material). B2 compares the post-trained model at maximum effort against open and closed models on reasoning and agent benchmarks, and Figure 7 tracks agent scores across RL training. How to read: higher is better except BPB; compare V4.1-Flash mainly with V4-Pro (≈3× more parameters) and with the frontier closed models (metrics: EM, Pass@1, Resolved, rating, BPB).

B1 — base models.

Benchmark	V4-Flash-Base	V4-Pro-Base	V4.1-Flash-Base
Total / activated parameters	284B / 13B	1.6T / 49B	552B / 8B–16B
MMLU-Pro (EM)	68.3	73.5	74.1
SuperGPQA (EM)	46.5	53.9	53.1
SimpleQA-Verified (EM)	30.1	55.2	42.3
HumanEval (Pass@1)	69.5	76.8	79.4
BigCodeBench (Pass@1)	56.8	59.2	60.6
MATH (EM)	57.4	64.5	61.1
MGSM (EM)	85.7	84.4	80.2
LongBench-V2 (EM)	44.7	51.5	45.2
BPB internal docs (lower is better)	0.617	0.590	0.564
BPB internal code (lower is better)	0.1562	0.1494	0.1443
BPB academic material (lower is better)	0.4929	0.4677	0.4305

B2 — post-trained models (maximum effort).

Benchmark	Opus-5	GPT-5.6 Sol	Kimi-K3	GLM-5.3	V4-Pro	V4-Flash	V4.1-Flash
GPQA Diamond	93.4	94.1	92.9	88.1	92.4	89.9	90.9
HLE, no tools	56.3	44.5	43.5	42.0†	42.7†	37.8†	36.8 (39.1†)
Codeforces rating	—	—	—	—	3348	3289	3471
Terminal-Bench 2.1	89.1	88.8	88.3	88.2	87.9	82.7	90.6
Terminal-Bench 3.0	43.3	34.4	17.7	28.3	11.8	7.6	30.0
Terminal-Bench 4.0	51.8	39.9	12.6	37.9	12.4	7.0	31.2
DeepSWE v1.1 (Resolved)	74.0	73.0	67.5	66.9	62.7	54.4	74.2
ProgramBench	37.0	23.0	17.5	19.0	15.5	—	20.3
CyberGym	—	84.5	80.0	84.5	83.3	76.7	88.1
ExploitGym	22.1	33.7	—	15.0	5.4	1.8	15.3
AutomationBench	50.3	45.8	46.7	48.8	43.2	37.7	54.8
Agents’ Last Exam	28.6	26.7	27.6	28.5	25.7	25.2	31.8
HLE with tools	63.6	—	59.8	62.5	60.0	51.5	63.9
BabyVision with tools	94.1	88.9	85.7	—	—	—	89.6

† text-only subset of HLE.

• Takeaway B1 — the base model matches a ≈3× larger predecessor. V4.1-Flash-Base is on par with V4-Pro-Base using about one-third of the parameters and one-third (decode) to one-sixth (prefill) of the activated parameters; it leads on MMLU-Pro, HumanEval and BigCodeBench and has the lowest BPB on all three internal corpora (3–8% below V4-Pro, 8–13% below V4-Flash). It trails V4-Pro on fact recall (SimpleQA 42.3 vs 55.2), long context (LongBench-V2 45.2 vs 51.5) and MATH, and scores lowest of the three on MGSM (multilingual grade-school math).
• Takeaway B2 — the post-trained agent leaps ahead of V4-Flash and tops many agent benchmarks. DeepSWE rises 54.4 → 74.2, Terminal-Bench 3.0 7.6 → 30.0, Terminal-Bench 4.0 7.0 → 31.2 and AutomationBench 37.7 → 54.8. V4.1-Flash posts the best score among all seven models on Terminal-Bench 2.1, DeepSWE, CyberGym, AutomationBench, Agents’ Last Exam and HLE with tools, and the best Codeforces rating among DeepSeek models (3471). RL keeps paying off: DeepSWE climbs from ≈57 to ≈72 over ≈2,000 RL steps, and extending training context to 1M lifts Terminal-Bench 3.0 to ≈29 (Figure 7).
• Takeaway B3 — a clear gap remains on the hardest tasks. Opus-5 leads by ≈20 points on Terminal-Bench 4.0 (51.8 vs 31.2) and ≈17 on ProgramBench (37.0 vs 20.3); V4.1-Flash has the lowest no-tool HLE score in the table (36.8 vs 56.3) and trails GPT-5.6 Sol on exploit writing (15.3 vs 33.7). On visual-agent tasks it beats Kimi-K3 but stays below both closed models.

Part C — Can users steer cost and deploy flexibly?

Question → setup. Three deployment questions. C1: does the effort level $b$ trade tokens for accuracy smoothly? The same checkpoint runs at $b$ from 25 to 100 on eight reasoning benchmarks (averaged), DeepSWE and Terminal-Bench 2.1 (Figure 9, Appendix Figures 11–12). C2: does performance survive a change of harness? The same checkpoint runs in 8 scaffold configurations at maximum effort, with 8 samples per DeepSWE task and 3 per Terminal-Bench task, 1M context and up to 500 model turns (Table 4). C3: do agent teams beat a single agent? A lead agent spawns teammates sharing one repository, compared with the strongest single-agent run at fixed wall-clock deadlines on 172 high-confidence ProgramBench tasks (Almost@1) and the no-GPU FrontierSWE v2 subset (Mean@5) (Figure 10). How to read: in C1 look for accuracy gained per extra token; in C2 a narrow spread across columns means robustness; in C3 the team column should exceed the single-agent column at every deadline.

C1 — reasoning effort (Pass@1).

Benchmark (harness)	$b=25$	$b=100$	Gain
8 reasoning benchmarks, average	67.1	76.3	+9.2
DeepSWE v1.1 (mini-SWE)	66.0	74.2	+8.2
Terminal-Bench 2.1 (DeepSeek Harness Minimal)	82.4	90.6	+8.2
MathArena Apex 2025	25.3	65.6	+40.3

C2 — harness transfer at maximum effort.

Harness	DeepSWE v1.1 (Resolved)	Terminal-Bench 2.1 (Pass@1)
Claude Code v2.1.251	69.8	88.0
Codex	65.6	84.1
OpenCode	65.5	85.0
Pi	66.2	86.1
mini-SWE	74.2	90.3
DeepSeek Harness Minimal	72.6	90.6
DeepSeek Harness Standard	70.5	85.8
DeepSeek Harness PTC	67.6	85.8

C3 — agent teams vs. a single agent.

Benchmark (metric)	Deadline	Single agent	Agent Team
ProgramBench, 172 tasks (Almost@1)	1 h	12.79	13.59
ProgramBench, 172 tasks (Almost@1)	8 h	20.39	30.04
FrontierSWE v2, no-GPU (Mean@5)	1 h	10.50	13.50
FrontierSWE v2, no-GPU (Mean@5)	20 h	28.20	32.90

• Takeaway C1 — effort is a usable cost dial with front-loaded gains. Raising $b$ from 25 to 100 adds 8–9 points on average at ≈2.5× the output tokens, and response length scales predictably (2.0–3.1× per benchmark). Efforts of 60–80 recover most of the accuracy at under half the tokens of $b=100$, whose final step lengthens agent trajectories 1.6–1.8× for small gains. Per-harness curves are noisier: in Claude Code, DeepSWE accuracy hovers ≈65–70 with no clear trend as effort rises.
• Takeaway C2 — skills transfer across harnesses, with a ≈9-point spread. DeepSWE ranges 65.5–74.2 and Terminal-Bench 2.1 ranges 84.1–90.6 across eight configurations. The best results come from mini-SWE and DeepSeek Harness Minimal, the configurations used for the headline numbers in Part B.
• Takeaway C3 — agent teams win at every deadline, and the margin grows with time on ProgramBench. The team’s lead over a single agent widens from +0.8 at 1 hour to +9.65 at 8 hours on ProgramBench, and reaches +4.7 at 20 hours on FrontierSWE v2. These results are preliminary and compare the strongest observed configurations.

---

5. Limitations

The central trade-off is asserted rather than isolated. The paper’s thesis is that aggressive compression costs almost nothing, yet no experiment isolates a single component: there is no run with FP8 instead of FP4 main KV, with per-layer instead of shared global KV, with exact instead of bounded SWA replay, or with standard instead of single-pass mHC; each is described as causing “negligible” or “marginal” degradation without numbers. At the same time, V4.1-Flash differs from V4-Flash in nearly everything else: twice the backbone (552B vs 284B), 196B Engram parameters, a new 45T multimodal corpus and a scaled-up post-training data pipeline. Its higher scores therefore cannot be credited to the compression, nor used to show that compression is free; the reader learns that the whole package is better, not what the 890-byte cache gives up.

Efficiency is reported in proxies, and some costs move elsewhere. Bytes per token and precision-weighted FLOPs stand in for what a deployer pays; the paper shows no throughput, latency, cache-hit rate, SWA-miss rate or price, and its 0.25 weight for FP4 operations itself favors the low-precision design. The “1/8 persistent cache” rests on SWA states occupying “nearly half” of V4’s persistent storage, a production statistic the paper does not report. The model is also larger, not smaller: ≈750B parameters (backbone plus Engram) must sit in memory, and decode activates 16B against V4-Flash’s 13B, so per-token decode compute is higher below ≈100K tokens. Bounded replay makes internal states, and hence outputs, depend on where a cache hit occurred, which complicates reproducibility across servers and creates exactly the untested boundary cases the authors flag in their own limitations.

The comparisons are hard to verify and partly favorable by construction. Several numbers come from internal benchmarks (the Codeforces set, the BPB corpora) or self-selected subsets (172 “golden” ProgramBench tasks, a no-GPU FrontierSWE subset), and competitor scores carry no stated provenance or harness. V4.1-Flash’s headline DeepSWE and Terminal-Bench 2.1 scores come from its best harnesses; in Claude Code it scores 69.8 and 88.0, below Opus-5’s listed 74.0 and 89.1. The text also overstates in places: Section 5.3 calls accuracy gains with effort monotonic while Appendix B.2 reports dips; the conclusion compares with “Fable-5 and GPT-6 Astra,” which appear in no table; and the claim of completing “over 95% of real-world tasks” has no measurement behind it. On the hardest tasks the gap to the frontier remains large (Terminal-Bench 4.0: 31.2 vs 51.8; no-tool HLE: 36.8 vs 56.3).

---

6. How to reproduce

Step 1 — obtain the model. The checkpoint is public at huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash, stored as FP8 safetensors under a dedicated deepseek_v41 architecture tag. At ≈1 byte per parameter, the ≈750B backbone-plus-Engram parameters need ≈750 GB of accelerator or host memory before any KV cache, i.e. a multi-GPU node; Engram tables can live in host memory and be prefetched. The model card’s usage snippets (Transformers, vLLM, SGLang) are generic Hub templates, so the first check is whether your engine version implements CED, the three CSA2 modes, the hierarchical indexer and the FP4 main KV cache.

Step 2 — serve it with the efficiency features. DeepSeek’s kernels are open source: FlashMLA (fused attention), DeepGEMM (Mega-MoE, Mega-mHC, FP8 GEMMs), TileKernels and DeepSelect (top-K selection), all under github.com/deepseek-ai. The paper does not identify a released serving stack for the system-level pieces: the persistent-cache manager with the short-lived DRAM pool for SWA states, encoder and decoder bounded replay, encoder–prefill–decode disaggregation and the DSpark verification scheduler. Without them you reproduce the model’s quality but not its memory and prefill savings; with them, the decoder replay of exactly the last 128 prompt tokens must match the post-training simulation.

Step 3 — verify the efficiency claims. The 890 bytes/token follows directly from the configuration (Section 3f, C2), and the decode-FLOP curve can be recomputed from layer counts and the precision weights (1, 0.5, 0.25). Measuring the persistent-cache ratio and the quality cost of bounded replay requires your own traffic: log cache hits, SWA misses and replay frequency, and compare outputs with exact versus bounded replay on the same prompts. The paper does not release its V4 cache statistics, so the “nearly half” SWA share must be re-measured.

Step 4 — reproduce the evaluations. DeepSeek Harness is open source under the MIT license at github.com/deepseek-ai/deepseek-harness; use Minimal mode (one bash tool) for Terminal-Bench, the mini_swe_v2 port of mini-swe-agent (commit 04d809ceab9d) for DeepSWE, and the scaffold versions in Appendix B.1 for Table 4. Settings: temperature 1.0, top-p 0.95 for agents (1.0 for reasoning), 1M context (512K for visual agents), at most 500 turns, 8 samples per DeepSWE task and 3 per Terminal-Bench task, no network, git history stripped and build caches purged. Set effort with the system-prompt line “Reasoning Effort: {b}” or the API tiers low/high/max = 50/75/100. Missing: the internal Codeforces set, the internal BPB corpora, the base-model prompt templates and evaluation framework, the list of 172 ProgramBench tasks and the FrontierSWE no-GPU subset, and the source and harness of every competitor score.

Step 5 — retrain (largely out of reach). Pre-training hyperparameters are complete (layer layout, widths, expert counts, learning-rate schedule, batch size, optimizer constants, context-extension point), and the components have companion papers: DeepSeek-V4 (arXiv:2606.19348), mHC (arXiv:2512.24880), Engram (arXiv:2601.07372) and DSpark (arXiv:2607.05147). Not released: the 45T-token corpus, the vision encoder’s 47B image–text pairs and 236B-token fine-tuning set, the synthesized tasks and environments, the DSec sandbox platform, the RL hyperparameters for effort control ($\kappa_0$, $\tau$, $C_{\max}$, $L_{\text{norm}}$, the set of training effort levels, samples per level), the agent-team reward weights, the model-merging recipe and the 40+ OPD teachers.