Reading notes on:

1. Executive Summary

Kimi K3 is a 2.8-trillion-parameter (3T-class) model with a 1M-token context window and native vision. While it trails proprietary models like Claude Fable 5 and GPT-5.6 Sol, it sets a new upper bound for open weights and shows a 2.5× improvement in scaling efficiency over Kimi K2. The focus is long-horizon agentic capability — coding, knowledge work, complex reasoning — in the same open-frontier lineage as LongCat-2.0, GLM-5.2, and Inkling.

Kimi K3 block flow: AttnRes residual routing across n-1/n-2/n-3, KDA, Gated MLA, Stable LatentMoE

2. Architecture & Structural Innovations

  • Attention (KDA & AttnRes): K3 abandons uniform accumulation across depth. Attention Residuals (AttnRes) selectively retrieve representations across model depth, paired with Kimi Delta Attention (KDA) as an efficient foundation for scaling attention and information flow across sequence length.
  • Block flow: residual routing spans multiple previous blocks ($n{-}1, n{-}2, n{-}3$), weighted by scalars ($\alpha_w$) across KDA, Stable LatentMoE, and Gated MLA modules, with normalization, linear maps, L2 convolutions, and activations.
  • MoE sparsity: a Stable LatentMoE framework with extreme sparsity — only 16 of 896 experts active per token. Contrast this with the milder 8/256 sparsity in GLM-5.2 and the mini-activation approach in MiniMax-M2.
  • Routing optimizations: at this sparsity, standard heuristics fail. K3 introduces Quantile Balancing (derives expert allocation directly from router-score quantiles, eliminating sensitive balancing hyperparameters), Per-Head Muon (extends the Muon optimizer to independently optimize attention heads — see SOAP), a Sigmoid Tanh Unit (SiTU) activation, and Gated MLA for attention selectivity.

3. Training, Quantization & Systems

  • Quantization-aware training from the SFT stage onward: MXFP4 weights, MXFP8 activations for hardware compatibility — the same low-precision-in-the-loop discipline explored in The 4-bitter Lesson: NVFP4 in the RL Loop.
  • Expert-parallel training: fully balanced with static shapes and zero host synchronization on the critical path to avoid throughput degradation from expert imbalance.
  • Inference & caching: KDA challenges conventional prefix caching, so the team built a custom prefill cache (contributed to vLLM), enabling competitive token pricing. Optimized for supernodes of 64+ accelerators.

4. Capabilities

K3’s agentic strength is shown on domain tasks rather than static benchmarks:

  • GPU compiler development: autonomously built MiniTriton — a tile-level IR over MLIR, optimization passes, and a PTX codegen pipeline — rivaling or beating Triton on some workloads and sustaining stable nanoGPT training.
  • Kernel optimization: optimized 512-head-dim MLA, KDA, and AttnRes kernels on H200 and alternative GPGPUs, outperforming GPT-5.6 Sol and Claude Opus 4.8. This is the model-writes-its-own-kernels frontier that abstractions like CODA are designed to enable.
  • Autonomous chip design: as an EDA agent, designed a nano-model chip on Nangate 45nm in 48 hours — 4 mm², timing closed at 100 MHz, 8,700 tokens/s decode, 1.46M standard cells, 0.277 MB SRAM, an INT4 MAC array with fused dequantization.
  • Scientific workflows: reproduced I–Love–Q universal relations in ~2 hours (a 1–2 week human task), synthesizing 20+ papers, 300+ equations of state, and 3,000+ lines of Python.
  • Multimodality: native text/image/video; multi-agent setups (20+ subagents over 391 gravitational-wave events) and autonomous video editing (motion-matched cuts, beat sync from 56 clips).

5. Limitations

  • Statefulness: K3 is highly sensitive to preserved “thinking history.” Switching models mid-session or using harnesses that drop historical context causes unstable generation — an operational hazard familiar from the harness-design discussion in Harness Engineering for Self-Improvement.
  • Excessive proactiveness: trained for long-horizon autonomy, K3 makes unprompted decisions under ambiguity; production use needs explicit constraints via system prompts or AGENTS.md.

Takeaway: the move from heuristic load balancing to Quantile Balancing is a major stability win at 3T scale, solving expert collapse in extreme (16/896) sparsity. Combined with K3’s ability to bootstrap low-level systems — PTX codegen, INT4 MAC arrays, 512-head MLA kernels — it marks an inflection point where open-weights AI can directly accelerate its own hardware and software lifecycle.