Reading notes on:

Thinking Machines Lab released Inkling, an open-weights Mixture-of-Experts transformer trained from scratch: 975B total / 41B active parameters per token, a 1M-token context window, pretrained on 45T tokens across text, images, audio, and video. It follows the interaction-first design philosophy the lab laid out in Native Interaction Models.

Inkling architecture: 5:1 attention interleave, 256+2 MoE with 6 active, and key departures

1. Architectural Departures

Inkling builds on the DeepSeek-V3 MoE framework (cf. DeepSeek-V4 Architecture & Training) with several long-context, inference-efficiency modifications:

  • Routing: 256 routed + 2 shared experts per MoE layer; exactly 6 routed experts active per token.
  • Auxiliary-loss-free balancing: a sigmoid-based router with a load-balancing bias; selected routed and shared outputs are normalized jointly, improving balance without complicating the loss landscape.
  • Attention interleaving: sliding-window and global layers at a 5:1 ratio, with 8 KV heads — heavily favoring local context.
  • Positional embeddings: breaking from RoPE, Inkling uses relative positional embeddings, which extrapolate significantly better to the 1M-token limit.
  • Short convolutions: applied after the key/value projections and on the attention/MLP residual-branch outputs before they rejoin the main stream, improving signal flow.

2. Encoder-Free Multimodality

Rather than bolting on vision/audio encoders, Inkling is encoder-free:

  • Audio: processed natively as dMel spectrograms.
  • Vision: images fragmented into 40×40 patches, encoded by a four-layer hMLP.

Both modalities pass through a lightweight embedding layer and are processed jointly with text tokens.

3. Optimization & Training Math

  • Hybrid optimizers: Muon for large matrix weights, Adam for other parameters — the same Muon momentum that features in DeepSeek-V4 and is contextualized against second-order methods in SOAP.
  • Weight decay coupled to LR²: $WD \propto LR^2$. This exact coupling keeps overall weight magnitude stable across a massive training horizon.

4. RL at Scale: Proper Scoring & Log-Linear Gains

Post-training used an SFT bootstrap on synthetic data, then massive asynchronous RL over 30M+ rollouts.

  • Epistemic calibration: RL driven by proper scoring rules over resolved real-world questions, plus abstention-aware rewards — the policy is optimized to hedge or say “I don’t know” when calibrated confidence is low.
  • Scaling laws: reasoning aggregates (AIME, HLE, GPQA) scaled strictly log-linearly across the full 30M+ RL run — an empirical echo of The Architecture of Scaling Laws.

5. Emergent Behaviors: CoT Compression & Controllable Effort

  • Controllable thinking effort: by adjusting per-token cost and the system message during RL rollouts, Inkling learned computational budgeting. At inference you can dial “effort,” letting it match Nemotron 3 Ultra using roughly a third of the tokens. The general mechanism is detailed in The Mechanics of Reasoning Effort and Inference Scaling.
  • Emergent CoT compression: with no explicit brevity reward, the chain-of-thought became telegraphic purely to save tokens:
    • Early RL (verbose): “We need to understand the operator. The 5D line element is $ds^2 = e^{2A(x)}(ds^2_{4d} + dx^2)$… So the internal metric is $e^{2A(x)}dx^2$.”
    • Late RL (telegraphic): “We need determine eigenvalue problem… Actually $ds^2 = e^{2A}[ds_4^2 + dx^2]$. So internal metric is $e^{2A}dx^2$; warp factor same for 4d and internal? Yes.”

    The model organically dropped articles and connectives while preserving rigorous derivations.

For edge deployments, Thinking Machines is also previewing Inkling-Small, a 12B-active (276B-total) MoE that trades peak capability for hyper-low latency on the same post-training stack.