Reading the following:

LongCat-2.0 is a step up from LongCat-Flash in almost every dimension: it scales the Sparse Mixture-of-Experts (MoE) design to 1.6 trillion total parameters (with roughly 48 billion active parameters per token), pre-trains on 35 trillion tokens, and pushes context length to 1M tokens. The most striking part is the substrate: pre-training and large-scale deployment run entirely on AI ASIC superpods, sidestepping the Nvidia GPU ecosystem. Below is a breakdown of the architectural, algorithmic, and infrastructural derivations that make it competitive for long-context and agentic workloads.

1. Architectural Innovations: Attention and Embeddings

LongCat-2.0 keeps the LongCat-Flash foundation (Zero-Computation Experts, ScMoE, MTP) and layers on two new ideas: a sparse-attention scheme and an N-gram embedding expansion.

LongCat Sparse Attention (LSA)

The authors observed that the fine-grained sparse attention in DeepSeek Sparse Attention (DSA) is bottlenecked by its Lightning Indexer — specifically output discontinuity and the quadratic cost of scoring every query against every key. LSA attacks this with three orthogonal mechanisms:

  • Hierarchical Indexing (HI) — killing the quadratic cost. A coarse-to-fine, two-stage scheme. First, a coarse recall uses block-level approximate scoring; second, fine-grained token selection is restricted to the recalled candidates only. This shrinks the candidate space processed per query from $O(L)$ toward the recalled block budget.
  • Streaming-aware Indexing (SI) — memory I/O optimization. SI reshapes the token-selection budget by combining hardware-aligned contiguous access with dynamic random selection, turning fragmented HBM fetches into predictable sequential reads. The result is coalesced access and high effective bandwidth.
  • Cross-Layer Indexing (CLI) — cost amortization. Attention saliency is empirically stable across adjacent layers, so a single indexing pass is distilled to serve several consecutive layers at inference. In the 3-step MTP module, step 1’s index set is reused directly for draft steps 2 and 3.

N-gram Embedding Scaling

To grow representational capacity outside the MoE dimension, LongCat-2.0 expands the embedding space ~100× via N-gram token combinations, setting N = 5 and allocating 135B parameters to the module. The scaling rationale is the interesting part:

  1. The MoE sparsity limit. With MoE sparsity already at ~97%, dumping an extra 135B into the experts yields negligible gains. Redirecting that same budget into N-gram embeddings pays off far more.
  2. The budget-proportion rule. Their derivations show N-gram parameters must be bounded: past ~50% of the total budget the advantage collapses, so LongCat-2.0 keeps the N-gram share under 10%.

Insight: shifting parameters from experts to N-gram embeddings reduces large-batch decoding memory I/O, which directly accelerates generation.

2. Scalable Infrastructure on AI ASICs

Pre-training a 1.6T model over 1M-token context on >50,000 AI ASICs demanded system-level guarantees.

  • 6D parallelism & superpods. Beyond standard 5D parallelism, the team added EMBP to parallelize the massive N-gram embeddings. Superpods of up to 48 machines use an internal high-bandwidth all-to-all fabric with external RoCE, yielding a 30% pre-training throughput gain. For 1M context, an all-gather-based Context Parallelism (CP) scheme scales past 512.
  • Numerical reliability. To bound floating-point error accumulation across tens of thousands of ASICs, foundational operators were reworked with a binary-tree segmented accumulation strategy, plus bit-flip detection on compute-intensive operators to catch hardware anomalies immediately.

3. Inference: PD-Disaggregated Serving

Serving 1.6T parameters at 1M context is bound by HBM capacity, HBM I/O, and interconnect bandwidth. LongCat-2.0 uses a Prefill–Decode (PD) disaggregated deployment to balance Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT):

  • Prefill nodes are limited by inter-node MoE dispatch/combine traffic, so they use multi-node chunked pipeline parallelism (CPP) to shrink the expert-parallel (EP) domain, layered with attention sequence parallelism (SP).
  • Decode nodes are bound by device memory and KV-cache I/O. They shard with KV-cache parallelism (KVP) and deploy a massive EP128. Because EP128 raises load-imbalance risk, an asynchronous Expert-Parallel Load Balancing (EPLB) computes placement statistics off the forward critical path.
  • Device-level optimization. Using explicit per-core ASIC control, ScMoE runs the dense and MoE branches fully in parallel. Super Kernels eliminate intra-kernel launch overhead, and weights are prefetched into L2 to hide I/O behind preceding compute.

4. Post-Training: The MOPD Architecture

To turn the foundation model into an agentic tool, LongCat-2.0 uses a Multi-Expert Post-Training (MOPD) architecture with three expert groups:

  1. Agent Experts — atomic capabilities: precise tool invocation, multi-turn API parameter parsing, and self-correction to avoid infinite loops.
  2. Reasoning Experts — math, STEM, and multi-hop reasoning, with adaptive compute scaling by problem difficulty.
  3. Interaction Experts — human alignment, instruction following, and factual hallucination suppression.

Concluding Insight

LongCat-2.0 is a masterclass in co-designing a model around hardware realities. Recognizing the diminishing returns of scaling MoE parameters past 97% sparsity, they redirected the budget into a 135B N-gram embedding space — improving quality while sidestepping the memory-I/O wall. And a smooth 35T-token run with no irrecoverable loss spikes on 50K+ ASICs is evidence that frontier-scale training is no longer tethered to the GPU monopoly. The payoff shows up on the benchmarks that matter for this design: SWE-bench Pro 59.5 for coding and FORTE 73.2 for agentic workflows.