Ever wonder what really happens when you type a prompt and an AI spits out a response? It's not magic—it's a precise dance of math and code.
The Simple Version
Think of an LLM as a super‑fast autocomplete that never stops. Each step it guesses the next word, adds it, and repeats until you decide to stop.
How It Actually Works
The inference loop
When you call a function like model.generate(prompt), the model does not magically output a whole sentence. Instead it enters a loop:
Inside the loop the model runs an inference, picks the highest‑probability token, appends it to the input, and feeds the new sequence back in.
Technical Deep Dive
Tokenization
Text is broken into sub‑word units called tokens. Each token maps to a vector that the transformer can process.
Probability distribution
The model outputs a probability for every possible next token. Developers can shape this distribution with temperature or top_p to control randomness.
Sampling
Instead of always picking the top token, sampling draws from the distribution, giving more varied outputs.
Why This Matters
Understanding the token‑by‑token loop helps developers debug strange outputs and fine‑tune performance. It also demystifies why LLMs can hallucinate or repeat themselves.
For educators, it provides a concrete example to show students how AI works under the hood, turning a black box into a transparent tool.
Key Takeaways
- The generation process is a repeated inference‑sample‑append cycle.
- Tokenization converts raw text into numbers the model can handle.
- Probability distributions control how deterministic or creative the output is.
- Tech like Transformers.js makes this loop fast enough for browsers.
- Parameters such as temperature and top_p let you adjust randomness.
- Seeing the loop in action clarifies common LLM quirks.