Reading notes on:

The serving bottleneck for a frontier model is no longer FLOPs. For a long-horizon agent that reads a repository, calls tools, and comes back twenty minutes later, the binding constraints are prefill compute on every new turn, HBM capacity for the live KV cache, and persistent storage for prompt state that has to survive across tool calls and restarts. A model that is cheap per generated token but expensive per ingested token is the wrong shape for that workload.

DeepSeek-V4.1-Flash is a direct attack on all three: a 552B-parameter multimodal MoE — 196B of that conditional memory via Engram — natively supporting up to 1,000,000 multimodal tokens. The headline number is 890 bytes per token of global KV in HBM — roughly a quarter of DeepSeek-V4-Flash’s 3,514 bytes/token and 1/437 of DeepSeek-V1’s 389,120. Persistent host/SSD state shrinks to 1/8 of V4-Flash. And the model activates 8B parameters per prefill token against 16B per decode token, which is the part I find most interesting: prefill and decode are no longer forced to share an activation budget.

None of that comes from one trick. It comes from four compressions stacked multiplicatively — a stack split, a three-mode sparse attention, FP4 quantization, and a replay scheme that lets local caches be thrown away — plus two infrastructure changes that make the whole thing trainable.

DeepSeek-V4.1-Flash architecture: causal encoder-decoder split, CSA2 layer modes, hierarchical candidate pool, and the resulting KV footprint

1. The Causal Encoder-Decoder

A standard Transformer with sequence length $N$ and $L$ layers pays $O(N \cdot L)$ during prefill, because every layer computes its own KV for every position. Following YoCo, CED splits the 40-layer network in half: a 20-layer Causal Encoder ($l \in [1, L/2]$) and a 20-layer Decoder ($l \in [L/2+1, L]$).

The decoder’s global attention does not compute KV from its own hidden states. Both the KV entries $C_l$ and the compression weights $Z_l$ are projected directly from the final encoder hidden state $H_{L/2}$:

\[C_l = H_{L/2} W_l^{KV}, \quad Z_l = H_{L/2} W_l^{Z} \quad \text{for } l > \frac{L}{2}\]

with layer-dependent $W_l^{KV}, W_l^{Z}$. For $N \gg n_{\text{win}}$, the prefill cost becomes

\[O\!\left(N \cdot \frac{L}{2} + n_{\text{win}} \cdot \frac{L}{2}\right) \approx O\!\left(\frac{NL}{2}\right)\]

Prefill compute is halved. Decoder queries still run through 20 full layers, so the representation depth used to read the cache is unchanged — only the writing side is collapsed.

This is the mechanism behind the asymmetric 8B/16B activation. Prefill only has to run the encoder half plus a KV projection; decode runs everything. Where prefill/decode disaggregation separates the two phases at the cluster level, CED separates them at the parameter level, which is strictly better for the agentic pattern where each turn re-ingests a large prompt for a handful of generated tokens.

2. CSA2: Three Modes, Three Multiplicative Savings

CSA2 is the successor to the Compressed Sparse Attention in DeepSeek-V4, which was itself built on the DeepSeek Sparse Attention of V3.2. It compresses along three axes at once: head latent compression, sequence-block compression by factor $m$, and cross-layer cache reuse.

The cross-layer axis is implemented by statically assigning each layer one of three modes:

  1. Full — computes its own main KV and indexer Q, projects indexer K directly from the main KV, scores the entire visible context, and emits fresh top-$k$ indices.
  2. Reindex — reuses main KV and indexer K from the preceding Full layer, but computes its own indexer Q to rescore that shared K, producing layer-specific top-$k$ indices.
  3. Reuse — reuses both the main KV and the top-$k$ selection from a preceding Full or Reindex layer, skipping the indexer entirely.

The layout:

  • Encoder (20 layers). Layers 1–2 are pure SWA. Layers 3–20 run CSA2 at $m = 2$, arranged as 3 identical 6-layer blocks of 1 Full + 5 Reuse.
  • Decoder (20 layers). CSA2 at $m = 1$, arranged as 5 blocks of 4. Block 1 is 1 Full + 3 Reuse; blocks 2–5 are 1 Reindex + 3 Reuse.

Count the indexers: 3 in the encoder, 5 in the decoder, for 40 layers. That is the same idea as IndexCache and GLM’s IndexShare, but pushed further — those schemes reuse indices while each layer still keeps its own KV, whereas CSA2’s Reuse mode drops the KV too.

The Reindex mode is the design decision worth dwelling on. Pure index reuse (GLM-5.2’s uniform interleaving) is cheap but forces deep layers to attend where a shallow layer decided to look. Pure recomputation is expensive. Reindex keeps the cache shared while letting the selection be layer-specific, which is the cheap half of the recomputation — the indexer Q projection — and drops the expensive half.

CSA2 also simplifies V4’s CSA: no overlapping block projections, no absolute positional embeddings during KV compression, and indexer K obtained as a linear projection of main KV rather than a separate compression path from hidden states. All three are kernel-efficiency moves, and the last one is what makes Reindex cheap in the first place.

3. The Hierarchical Sparse Indexer

Reindex mode has an obvious scaling problem. If a Reindex layer rescores the shared indexer K across all visible positions, it pays $O(N)$ per decoded token — and at $N = 10^6$ with four Reindex layers, that cost dominates everything CSA2 just saved.

The fix is a bounded search domain, constructed once per token by the decoder’s first Full layer:

  1. That layer scores all causally visible positions and emits its top-512 token indices as usual. Simultaneously, it scores each 8-position block by the maximum token score inside it and takes the top 2,048 blocks.
  2. $2{,}048 \times 8 = 16{,}384$ positions become the shared candidate pool.
  3. Every downstream Reindex layer scores only inside that pool.

The indexer cost for all deeper layers goes from $O(N)$ to $O(1)$ in context length. The pool size is a constant, so at 4K context it is nearly the whole sequence and at 1M context it is 1.6% of it — which is why single-token decode FLOPs stay flat from 4K to 1M.

The block-max scoring is doing real work here. Taking the top 2,048 blocks by max-token-score rather than the top 16,384 tokens is a coarse-to-fine filter: it keeps neighbourhoods, which is what a deeper layer with a different query is likely to want, rather than a scattered set of positions that only the first layer’s query liked. The hierarchy is the point, not just the truncation.

4. FP4 Main KV, Justified by a Norm Bound

The main KV cache is compressed to MXFP4 — E2M1 elements with one E4M3 scale per 16 channels — via quantization-aware training, in the same family as the FP4 QAT in DeepSeek-V4’s infra work and the QAT/QAD line from NVFP4 training and quantization-aware distillation.

The interesting choice is what they removed: NVFP4’s second-level global scale. Dropping it simplifies the cache layout — no per-tensor scalar to fetch and apply — but only works if the values are bounded a priori. The argument:

  1. The largest trained RMSNorm weight magnitude in the backbone is $\approx 1$.
  2. For a 512-channel KV latent, RMS normalization bounds the $L_2$ norm by $\sqrt{512}$.
  3. RoPE is an orthogonal transformation, so it preserves that norm.
  4. Therefore the post-RoPE maximum absolute value across channels satisfies
\[\max_i |v_i| \le \sqrt{512} \approx 22.6\]

Observed maxima during training were $\approx 10$. The MXFP4 representable range is $448 \times 6 = 2688$ (E4M3 scale times E2M1 element), so there are two orders of magnitude of headroom. The global scale was insurance against a risk the architecture already rules out.

I like this derivation because it is the rare quantization argument that is structural rather than empirical. RMSNorm plus orthogonal RoPE is a hard bound, not a percentile from a calibration set — which means it holds for inputs the calibration set never saw. Against FP8, this halves global KV in both HBM and SSD.

5. SWA Bounded Replay: Don’t Persist Local State

Exactly reconstructing sliding-window attention state across $L$ layers requires replaying $L \times n_{\text{win}}$ tokens, because each layer’s window depends on the layer below it over a widening span. That is why SWA KV normally gets persisted rather than recomputed — and persisting it across multi-turn sessions is a large SSD bill.

SWA bounded replay simply declines to be exact. On a cache miss it replays only the most recent $n_{\text{win}} = 128$ tokens, truncating each layer’s window to the replay segment $[\max(s, i - n_{\text{win}} + 1),\, i]$. Early positions in the replay segment see a shorter window than they would have during the original forward pass; the report’s position is that the quality cost is negligible.

Two consequences:

  • Encoder. SWA KV can be dropped from persistent storage entirely. It lives in a short-lived host DRAM pool (10% allocation, minute-scale TTL), and DRAM misses are recovered by bounded replay with a minor prefill penalty.
  • Decoder. Decoder prefill forward passes are restricted to $n_{\text{win}}$ tokens, halving decoder prefill latency.

The 1/8 persistent footprint is these two factors multiplied:

  • Factor 1 — drop SWA KV: ×1/2. In V4, local SWA KV was roughly half of all persistent KV on SSD. Bounded replay means it is never written there, which halves persistent storage on its own.
  • Factor 2 — compress what’s left: ×1/4. The remainder is exclusively global KV, and CSA2’s cross-layer reuse plus MXFP4 shrink that to a quarter of V4-Flash’s global footprint.

$\frac{1}{2} \times \frac{1}{4} = \frac{1}{8}$. The two factors are independent by construction — one removes a tier of the cache, the other compresses the tier that survives — which is why they compose cleanly instead of competing for the same bytes.

6. Single-Pass mHC

Manifold-Constrained Hyper-Connections widen the residual stream into $n$ parallel streams $X_l \in \mathbb{R}^{n \times d}$, with token-wise predicted mixing coefficients:

\[X_{l+1} = B_l X_l + C_l \mathcal{F}_l(A_l X_l), \quad (A_l, B_l, C_l) = \mathcal{H}(X_l)\]

where $A_l \in \mathbb{R}^{1 \times n}$, $C_l \in \mathbb{R}^{n \times 1}$, $B_l \in \mathbb{R}^{n \times n}$. The problem is the data dependency: $A_l$ is predicted from $X_l$ and then applied to $X_l$, which forces three sequential kernels per block (residual update, coefficient prediction, input mixing) and $(4n+4)d$ of activation traffic — twice the theoretical lower bound.

The fix is a one-block shift:

\[X_{l+1} = B_l X_l + C_l \mathcal{F}_l(A_{l-1} X_l), \quad (A_l, B_l, C_l) = \mathcal{H}(X_l)\]

Input mixing now uses $A_{l-1}$, computed by the previous block. So each tile of $X_l$ can be read once to do two things at the same time: perform the current block’s input mixing, and accumulate the reduction statistics needed to predict $(A_l, B_l, C_l)$ for the next block. The fused Mega-mHC kernel hits the $(2n+2)d$ lower bound — a 50% cut in activation memory bandwidth.

This is the same species of move as flash attention’s online softmax or the coefficient staleness in async optimizers: a dependency that looked essential turns out to be shiftable by one step at negligible modelling cost, and the shift buys a fusion. It is worth noting how narrow the change is — $\mathcal{H}(X_l)$ still sees $X_l$; only the consumer of $A$ moves.

7. Optimizers: Head-Wise Muon and Sinkhorn Balancing

Two changes, both about matching update geometry to parameter structure.

Head-wise Muon. Linear weights use Muon, continuing from V4 and the broader higher-order optimizer line. The refinement is to split Q and K weight matrices by attention head before the preconditioned gradient orthogonalization. Muon’s orthogonalization treats its input as one matrix with one spectrum; a concatenated multi-head Q matrix is really a stack of heterogeneous blocks, and orthogonalizing across them lets a high-norm head dictate the conditioning of the rest.

Sinkhorn-balanced updates. Adam’s second-moment buffer is unaffordable for the 196B Engram table, the vocabulary embeddings, and the prediction heads. The replacement combines Nesterov momentum with Sinkhorn normalization:

Input: W_t, G_t, M_{t-1}, beta, eta_t, gamma = 0.18, K = 11

1: M_t <- beta * M_{t-1} + (1 - beta) * G_t
2: G_t <- beta * M_t + (1 - beta) * G_t            # Nesterov
3: mask rows with row_norm <= tau * mean_row_norm  # near-zero rows
4: U^(0) <- G_t
5: for k = 1..K:
6:     k odd :  U^(k)_{i,:} <- U^(k-1)_{i,:} / (||U^(k-1)_{i,:}||_2 + eps)
7:     k even:  U^(k)_{:,j} <- U^(k-1)_{:,j} / (||U^(k-1)_{:,j}||_2 + eps)
8: Delta_t <- sqrt(n) * U^(K)
9: W_{t+1} <- W_t - (gamma * eta_t) * Delta_t

Alternating row and column normalization is Sinkhorn iteration: it converges to diagonal scalings $D_r, D_c$ with

\[\Delta_t = \sqrt{n}\, U^{(K)} = \sqrt{n}\, D_r G_t D_c\] \[\frac{1}{n}\sum_{j=1}^{n} (\Delta_t)_{ij}^2 \approx 1, \qquad \frac{1}{m}\sum_{i=1}^{m} (\Delta_t)_{ij}^2 \approx 1\]

so both row RMS and column RMS are $\approx 1$. That equalizes update magnitude across features and across tokens, which is exactly the property Adam’s second moment buys — except this derives it from the current gradient instead of storing state, cutting optimizer memory for embedding parameters by 50% while matching Adam’s convergence.

The row masking on line 3 matters more than it looks: on a sparse embedding table most rows have near-zero gradient at any step, and Sinkhorn would happily normalize numerical noise up to unit norm. The mask is what keeps the balancing from amplifying nothing into something. This pairs naturally with Engram’s deterministic addressing — the touched rows are known from the token sequence.

8. Reasoning Effort as a Scalar

Post-training conditions the model on an explicit scalar effort value $b$ prepended to the system prompt, and shapes it with a length penalty subtracted from the trajectory reward:

\[r_{\text{len}}^{b,j} = -\min\!\left(C_{\text{max}},\; k(b)\,\frac{\ell_{b,j}}{L_{\text{norm}}}\right), \qquad k(b) = k_0 \exp\!\left(-\frac{b - b_{\min}}{\tau}\right)\]

with $\ell_{b,j}$ the generated token count, $L_{\text{norm}}$ a reference length, and $\tau = \lambda \Delta b$.

Why exponential decay in $k(b)$ is the right choice falls out of a short derivation. Let $p_x(\ell)$ be the probability of solving problem $x$ with $\ell$ reasoning tokens. The model optimizes

\[\ell_x^*(b) = \arg\max_{\ell \ge 0}\left[p_x(\ell) - k(b)\frac{\ell}{L_{\text{norm}}}\right]\]

with interior first-order condition $p_x’(\ell_x^*(b)) = k(b)/L_{\text{norm}}$. Assume marginal accuracy gains decay exponentially, $p_x’(\ell) \approx a_x \exp(-\ell/s_x)$:

\[a_x \exp\!\left(-\frac{\ell_x^*(b)}{s_x}\right) = \frac{k_0 \exp\!\left(-\frac{b - b_{\min}}{\tau}\right)}{L_{\text{norm}}}\]

Take logs:

\[\log a_x - \frac{\ell_x^*(b)}{s_x} = \log k_0 - \frac{b - b_{\min}}{\tau} - \log L_{\text{norm}}\] \[\ell_x^*(b) \approx C_x - s_x \log k_0 + \frac{s_x}{\tau}\,(b - b_{\min}), \qquad C_x = s_x \log(a_x L_{\text{norm}})\]

An exponential penalty schedule induces an affine relationship between requested effort and optimal reasoning length. The two exponentials cancel in log space, which is the whole trick — a linear penalty schedule would give you a logarithmic response and a useless dial at one end.

Note the per-problem slope $s_x/\tau$: harder problems (larger $s_x$, slower-decaying marginal returns) respond more to the same increment of $b$. The dial is not a global token cap; it is a shadow price on thinking, and the model spends more where thinking is worth more. API presets map to Low ($b=50$), High ($b=75$), Max ($b=100$). Compare the discrete-checkpoint and budget-forcing approaches surveyed in the mechanics of reasoning effort — a single continuously-conditioned model is a much cheaper thing to serve.

9. Benchmarks

Category Benchmark V4-Flash V4-Pro V4.1-Flash
Parameters activated / total 13B / 284B 49B / 1.6T 8B prefill, 16B decode / 552B
Global KV bytes per token 3,514 — 890
Reasoning Codeforces rating 3289 3348 3471
  MathArena Apex 58.6% 65.3% 65.6%
  GPQA Diamond 89.9% 92.4% 90.9%
Agentic DeepSWE v1.1 54.4% 62.7% 74.2%
  Terminal-Bench 2.1 82.7% 87.9% 90.6%
  CyberGym 76.7% 83.3% 88.1%
  AutomationBench 37.7% 43.2% 54.8%

External comparisons: MathArena Apex ties Kimi-K3 at 65.6%; DeepSWE v1.1 edges Opus-5 (74.0%) and GPT-5.6 Sol (73.0%); AutomationBench clears Opus-5 (50.3%) and GLM-5.3 (48.8%).

The shape of the result is what matters. V4.1-Flash trails V4-Pro on GPQA Diamond (90.9% vs 92.4%) — a knowledge-heavy single-shot benchmark where a 49B-activated model should win — while beating it by 11.5 points on DeepSWE and 11.6 points on AutomationBench. Long-horizon agentic benchmarks are dominated by how much context you can afford to keep and re-ingest, not by peak per-token capability. A model with a 4× smaller KV footprint and half the prefill cost can hold more of the task in view for the same serving budget, and on these benchmarks that buys more than raw activated parameters do.

One caveat on the table: V4-Pro’s KV footprint is not reported, so the 4× claim is specifically against V4-Flash.

Takeaways

  1. Decouple prefill and decode activation. CED’s 8B/16B split targets the actual agentic cost structure: many ingested tokens, few generated ones. Halving prefill FLOPs while keeping full decoder depth for queries is a better trade than shrinking the model uniformly.
  2. Layer-dimension reuse beats head-dimension reduction. Block compression at $m=2$, cross-layer main-KV sharing, and top-$k$ index reuse multiply together. Eight indexers across 40 layers is a far bigger lever than shaving head dims.
  3. Bound the search, not just the cache. Sparse attention’s indexer becomes the bottleneck at 1M tokens. A 16,384-position candidate pool built by block-max scoring converts every deep layer’s indexing from $O(N)$ to $O(1)$, and is what actually makes decode FLOPs flat across context length.
  4. Structural bounds beat calibration ranges. RMSNorm plus orthogonal RoPE bounds post-RoPE latents by $\sqrt{512}$, so NVFP4’s global scale can be dropped rather than merely tuned away. Quantization arguments that follow from the architecture generalize; ones that follow from a calibration set do not.
  5. Local state should be recomputed, not persisted. Storing SWA KV on SSD across sessions is paying storage forever to avoid 128 tokens of recompute — and in V4 that tier was half the persistent bill. Bounded replay plus a short-TTL DRAM tier inverts it, and composes with the 4× global-KV shrink to reach 1/8.
  6. Shift a dependency by one block and a fusion appears. Single-pass mHC gives up nothing that matters and hits the $(2n+2)d$ traffic lower bound.
  7. An exponential penalty schedule makes effort a linear dial. One continuously-conditioned checkpoint replaces a family of reasoning-length variants, and the per-problem slope $s_x/\tau$ means the dial automatically spends more on hard problems.

The through-line is that every one of these is a system decision justified by an inequality. The KV number is not an ablation result; it is what falls out when you split the stack, share across layers, bound the indexer, quantize under a norm bound, and refuse to persist what you can cheaply rebuild. Each step is individually modest and they compose to 437×.