How Meituan’s LongCat 2.0 Trains a 1.6T MoE Model Without…

Meituan’s LongCat 2.0 shows a 1.6‑trillion‑parameter MoE model can be trained without NVIDIA GPUs or TPUs, using n‑grams, sparse attention and custom ASICs to cut cost and hardware reliance.

Meituan just proved you can train a 1.6‑trillion‑parameter model without NVIDIA GPUs or TPUs.

⚡ TL;DR: Meituan’s LongCat 2.0 demonstrates a 1.6T MoE model can be trained without NVIDIA GPUs or TPUs, using n‑grams, sparse attention and custom ASICs to slash cost and hardware reliance.

The Simple Version

Think of a massive library. Instead of buying more shelves (more experts), you add smarter index cards (n‑grams) that point to the right books faster.

How It Actually Works

N‑gram Power

LongCat swaps extra expert parameters for dense n‑gram embeddings. These embeddings capture context cheaply, giving more corpus knowledge per parameter.

🎯 Think of it this way: Adding n‑grams is like adding a QR code that instantly tells you which shelf a book lives on, instead of building a whole new shelf.

Sparse Attention Tricks

The model uses a modified sparse‑attention pattern inspired by DeepSeek. Memory access becomes predictable, letting the “helper” layers cache results across the whole context.

🎯 Think of it this way: It’s like a librarian who knows exactly which aisle a patron will need next, so they only pull the relevant books instead of scanning the whole library.

Speculative Decoding

A lightweight “draft picker” model proposes token sequences that the main model verifies. This speeds up generation while keeping quality high.

Technical Deep Dive

LongCat 2.0 is a 1.6‑trillion‑parameter Mixture‑of‑Experts (MoE) model trained on over 50,000 chips for 35 trillion tokens.

Training avoided NVIDIA GPUs and Google TPUs by using custom ASICs tuned for prefill versus decode workloads.

The architecture mixes expert routing with n‑gram embeddings, reducing the need for additional expert layers and hitting diminishing returns.

Long‑context cost is lowered through coarse‑to‑fine selection and caching, making extended context affordable.

Why This Matters

Reducing reliance on NVIDIA’s hardware and CUDA stack lets companies build large models on cheaper, alternative silicon.

Open‑sourcing the model encourages broader research into efficient MoE training and alternative attention schemes.

Speculative decoding and custom ASICs can cut inference latency, making real‑time long‑context tasks feasible.

These advances push the industry toward more sustainable, cost‑effective AI development.

Key Takeaways

1.6 T parameters can be trained without NVIDIA GPUs or TPUs.
N‑grams provide cheap context information, replacing extra expert parameters.
Sparse attention with predictable memory access cuts helper overhead.
Custom ASICs optimize prefill vs decode workloads.
Speculative decoding speeds up token generation.
Open‑source models accelerate industry‑wide efficiency research.
Lower hardware barriers enable cheaper, larger AI experiments.

How Meituan’s LongCat 2.0 Trains a 1.6T MoE Model Without NVIDIA GPUs or TPUs