Training Agents 2: Master Model Distillation for Custom AI…

Explore model distillation techniques — off‑policy, on‑policy, and self‑distillation — to shrink AI models while preserving performance.

Ever wanted to shrink a massive AI model without losing its smarts? This live tutorial shows exactly how.

⚡ TL;DR: Learn the four axes of distillation — signal, data source, timing, teacher identity — and how to apply off‑policy, on‑policy, and self‑distillation in practice.

The Simple Version

Think of distillation like a master teacher handing notes to a bright student. The student learns the essential knowledge in a smaller notebook.

How It Actually Works

Signal & Data Source

The “signal” is the guidance the teacher provides, while the “data source” decides what examples the student sees.

🎯 Think of it this way: The teacher writes answer keys; the student practices with those keys.

Timing & Teacher Identity

When you collect the data (off‑policy) versus scoring the student live (on‑policy) changes the flow. The “teacher identity” can be an open‑weight model or a black‑box API.

🎯 Think of it this way: One method copies a library of books; the other checks each answer as you write.

Distillation as RL

KD distance acts as a dense reward, shaping the student’s predictions token by token.

Technical Deep Dive

In the repo, distillation runs inside TRL’s training loop, using Trainer with a custom loss that adds a KL term.

Off‑policy steps: generate teacher outputs, store them, then run supervised fine‑tuning (SFT) on the student.

On‑policy steps: sample student outputs, query the teacher for a score, and update with gradient ascent on the reward.

Self‑distillation loops the student as its own teacher, often matching a larger model’s performance after a few epochs.

Why This Matters

Distillation lets you deploy powerful AI on edge devices, cut inference costs, and iterate faster on custom agents.

Enterprises can fine‑tune models for specific tasks without the budget of training from scratch.

Researchers gain a scalable way to explore new agent behaviors while staying within compute budgets.

Key Takeaways

Four axes — signal, data source, timing, teacher identity — organize every distillation strategy.
Off‑policy uses pre‑generated teacher data; on‑policy scores the student in real time.
Self‑distillation can rival a stronger teacher when the model stabilizes.
KD distance provides a token‑level reward, turning distillation into a reinforcement signal.
All methods run inside Hugging Face TRL, making experimentation reproducible.
Distilled models are smaller, cheaper, and still retain most of the teacher’s capability.
The repo includes full code for each pipeline, ready to clone and run.

Training Agents 2: Master Model Distillation for Custom AI Agents