Imagine if your music app didn't just guess what you liked based on a hidden algorithm, but could actually have a conversation with you about your mood, your goals, and your mistakes. Spotify is moving away from the 'black box' of recommendations toward something they call generative personalization.
The Simple Version
For years, music recommendations have been like a vending machine: you press a button (or the app predicts one for you), and a playlist pops out. You don't know why the machine chose those songs, and you can't really argue with it.
Spotify is replacing the vending machine with a world-class librarian. This librarian doesn't just know where the books are; they know your entire history, they understand the nuances of 'moody rain-day jazz,' and they allow you to correct them in real-time.
How It Actually Works
To make this happen, Spotify had to teach Large Language Models (LLMs) to 'speak' music. LLMs are great at words, but they don't naturally understand a catalog of 100 million tracks.
The Vocabulary of Music: Semantic IDs
Spotify uses semantic IDs to bridge the gap between text and audio. They take existing content embeddings—mathematical representations of a song or podcast—and use a quantization algorithm to turn those complex vectors into discrete tokens.
Essentially, they created a new alphabet where a specific token represents a specific piece of content. These tokens are then added to the vocabulary of an open-weight LLM, such as Qwen or Llama.
The NEO Training Recipe
Training these models isn't a single step; it's a four-stage process Spotify calls NEO. This structured approach ensures the model learns the music catalog without forgetting how to speak human languages.
- Semantic Foundation: The semantic ID tokens are constructed and added to the LLM's vocabulary.
- Domain Grounding: The model learns the mapping between these IDs and natural language. Crucially, Spotify freezes the LLM backbone during this stage, training only the new embeddings.
- Capability Induction: The model undergoes multitask instruction tuning. Here, the entire model is unfrozen to learn specific Spotify tasks like retrieval and recommendation.
- Post-Training: An optional stage involving techniques like RL (Reinforcement Learning) fine-tuning to polish performance.
Steerability and Reasoning
This architecture allows for 'steerable' experiences. As Yves Raimond puts it, Spotify is moving from "personalization as guessing... to personalization as reasoning."
This is visible in features like the Spotify DJ, where users can tap a button to steer the session in a new direction, or Prompted Playlists, where users can request hyper-specific vibes, such as bands playing in San Francisco on a specific night.
Technical Deep Dive
From an architectural standpoint, the most critical decision in the NEO recipe is the use of a frozen backbone during domain grounding. In traditional 'continued pre-training,' developers often feed domain-specific data into a model and update all weights. However, Spotify found that this often 'wiped out' the model's core language abilities—a phenomenon known as catastrophic forgetting.
By freezing the backbone and only training the embedding layer for the semantic IDs, they maintain the LLM's reasoning capabilities while grounding it in the Spotify catalog. This allows the model to handle complex, ambiguous queries by reasoning through the user's intent before selecting the appropriate semantic IDs.
Spotify also tackled the cold-start problem—the difficulty of recommending new items (like audiobooks) that have very little user interaction data. Through multitask instruction tuning, the model learned to transfer knowledge from data-rich domains (podcasts) to data-poor domains (audiobooks). This 'cross-learning' allowed the model to make meaningful audiobook recommendations even without a deep history of user behavior for those specific titles.
On the inference side, the team experimented with different decoding strategies. While Top-P sampling is common for creative writing, Spotify found it significantly hurt recommendation accuracy. They opted for beam search, which explores multiple high-probability paths to find the most accurate sequence of IDs. Remarkably, they found that 98% of generated semantic IDs were valid even without using constrained decoding, though they still use it for specific content targeting to reduce latency overhead.
Why This Matters
The shift to a 'Large Taste Model' fundamentally changes the relationship between the user and the algorithm. By introducing a Taste Profile, Spotify is making the recommendation engine transparent. Users can now see what the AI thinks they like and explicitly edit it—for example, telling the AI that Disney songs are for the kids and should not influence their personal recommendations.
Beyond user control, this system is designed to break 'habitual listening.' By using LLM judges grounded in actual user behavioral signals, Spotify can identify when a user is stuck in a loop and suggest unfamiliar content that still aligns with their deeper tastes. This approach increased alignment with human preferences by 91% on ambiguous queries.
Ultimately, this represents a transition from a retrieval system to a generative system. Spotify is no longer just finding a song that fits a profile; they are generating an entire audio experience—including personal podcasts and daily briefs—that is dynamically shaped by the user's real-time input.