How Uber Built Closed-Loop Evals for Its Multimodal…

Uber's AI team created a closed-loop evaluation system for multimodal food-photo edits, balancing brand fidelity, safety, and creativity while avoiding reward hacking.

Ever wondered how Uber makes food photos pop without losing a restaurant's unique style?

⚡ TL;DR: Uber built a closed-loop evaluation system for its food-enhancement AI, balancing brand fidelity, merchant safety, and creativity while fighting reward hacking.

The Simple Version

Think of a photo editor that knows each restaurant's look and never copies another.

How It Actually Works

🎯 Think of it this way: It's like a bouncer who checks every guest's outfit to match the club's vibe.

Reward Design

They trained a reward model to score image edits on realism and brand consistency.

🎯 Think of it this way: The model gives points only when the dish looks tasty and still wears the chef's signature logo.

Closed Feedback Loop

Offline metrics flagged odd outputs, while online A/B tests measured real clicks.

🎯 Think of it this way: Like a rehearsal that catches mistakes before the show, then tweaks the script live.

Safety Guardrails

Strict rules prevented the AI from altering packaging or colors.

Technical Deep Dive

They used a multimodal transformer to encode dish images and text menus.

Reward scores combined CLIP similarity with a brand-logo detector.

Offline evals ran on a held-out dataset; online evals fed back conversion rates.

Guardrails enforced a whitelist of allowed transformations.

Why This Matters

Small merchants rely on authentic visuals to stand out.

Homogenized images could hurt local brand identity.

Uber's approach shows how to safely scale creative AI in tight domains.

Key Takeaways

Align rewards with both visual quality and brand rules.
Mix offline metrics with live user signals.
Build explicit safety filters before deployment.
Iterate quickly using closed-loop feedback.
Use multimodal models that understand text and image together.
Guardrails prevent unwanted homogenization.
Scalable evals are essential for niche AI products.

How Uber Built Closed-Loop Evals for Its Multimodal Food-Enhancement Agent