Beyond Prompting: Paul Bakaus on Harness Engineering

By LatestLLM Editorial ·

Beyond Prompting: Paul Bakaus on Harness Engineering

Move past 'AI slop' by mastering harness engineering. Learn Paul Bakaus's techniques for adversarial sub-agents, inverted judges, and model-specific builds.

Key takeaways

  • Prompting is the 'floor', while harness engineering is the 'ceiling' of AI capability.
  • Banning specific elements doesn't create originality; it just moves the model to the next nearest cluster in latent space.
  • Adversarial sub-agents and 'inverted judges' prevent the model from grading its own homework or falling into 'maximalist' design traps.
  • Using standard output for instructions improves following but breaks prompt caching.
  • Models are overfitted differently (e.g., Gemini's animation obsession), requiring model-specific harness builds and compilers.
  • Context pollution in protocols like MCP can degrade model performance by cluttering the prompt window.

Ever wonder why every AI-generated website looks like a variation of the same 'modern' template—lots of white space, a few purple gradients, and that specific shade of sterile beige? Paul Bakaus, the creator of the Impeccable skill, calls this 'AI slop,' and he argues that simply telling the AI to 'be creative' is a losing battle.

⚡ TL;DR: Paul Bakaus argues that prompting is only the 'floor' of AI capability. To avoid generic 'AI slop,' he uses 'harness engineering'—a system of adversarial sub-agents, inverted judges, and model-specific compilers—to force LLMs out of their predictable latent space clusters and into professional-grade output.

The Simple Version

Imagine you're asking a student to write a poem. If you just say 'write a poem,' they'll give you the most average, cliché version possible to ensure they don't fail. This is what Bakaus calls 'the model's gravity'—the AI's tendency to drift toward the most probable, median answer found in its training data.

If you tell the student, 'Don't use the word love,' they won't suddenly become Shakespeare; they'll just use the word 'affection.' They are still playing it safe, just in a slightly different cluster of clichés.

Bakaus's solution isn't to give better instructions (prompting), but to change the environment the student works in (harnessing). Instead of one student, he creates a system where one person writes, another critiques, and a third provides a random, weird inspiration that forces the writer to think differently.

How It Actually Works

Fighting the 'Gravity' of the Median

When we prompt an LLM, it navigates a latent space—a mathematical map of concepts. Most prompts lead the model toward the center of a cluster. For design, this results in 'Claude beige' or overused fonts like Inter and Roboto.

🎯 Think of it this way: An LLM is like a river. Prompting is trying to steer the boat with a paddle, but the current (the model's gravity) always pulls you toward the center of the channel. Harness engineering is like building a series of dams and locks to force the water into entirely new directions.

To break this, Bakaus uses 'anti-attractors.' One technique is 'shaving the safe picks': he asks the model to name its top three most likely font choices, and then explicitly tells it to throw all three away. This forces the model to look further away in its latent space for the fourth or fifth option, which is often where the actual creativity lives.

Adversarial Sub-Agents and Blindness

A major problem in AI workflows is that models are often too agreeable. As Bakaus puts it, "a single thread grading its own work always says it did well." This is essentially the AI grading its own homework.

To solve this, Impeccable uses two 'blind' sub-agents that cannot see each other's work. One acts as a Design Director, focusing on hierarchy and aesthetics. The other is a Deterministic Linter, a script that checks for hard rules like color contrast or font limits.

Because the sub-agents are isolated, they can't anchor on each other's biases. As Bakaus notes, "Two blind opinions beat one confident guess." Only after both have finished does a main synthesis thread combine their feedback.

The Inverted Judge

Bakaus observed that models are often 'maximalist'—they believe that 'more is more.' In a design context, this means the AI often over-decorates or adds unnecessary complexity, which it then rates highly because it perceives the effort as quality.

To counter this, he sometimes builds judges that invert the response of the model. If the AI judge rates a design very high, the system flags it as potentially 'bad design' because it's likely just falling into the maximalist trap. This heuristic helps strip away the fluff and move closer to a professional, minimalist aesthetic.

Technical Deep Dive

From a technical architecture perspective, Bakaus is moving the logic from the probabilistic (the LLM's guess) to the deterministic (code-driven constraints). This is a critical shift for production-grade AI tools.

One powerful method is injecting instructions via standard output from a script rather than burying them in prose. Bakaus found that "when you put something out... from the standard out of a script... the model will follow it a lot more than before." While this is highly effective for instruction following, he notes a major trade-off: it breaks prompt caching because the input becomes dynamic and unique every time.

Furthermore, Bakaus treats his skill like a Mixture of Experts (MoE). Instead of one massive, messy system prompt, he routes requests to specific Markdown (.md) files. This prevents 'instruction drift,' where a model ignores certain rules because the prompt is too bloated.

He also addresses the reality that every model is overfitted differently. For example, he notes that Gemini "loves to animate pictures," often adding unwanted hover effects. Codex, on the other hand, has a strange preference for "bad letter spacing" and "extremely rounded borders." To combat this, his system generates model-specific builds. If the model is Codex, it loads a specific set of rules—including 'gates' to prevent it from compressing instructions—that are not loaded for Claude or GPT-4o.

The Distribution Struggle and Context Pollution

Scaling these skills introduces a 'works on my machine' problem. Many shared skills only work on the specific model the author used. To solve this, Bakaus developed a custom installer and compiler that can compile a skill for every harness using substitution techniques.

He expresses skepticism toward the Model Context Protocol (MCP) due to context pollution. For context, context pollution occurs when a tool injects too much metadata or irrelevant instructions into the prompt window, which can distract the model and degrade its reasoning capabilities.

Regarding distribution, Bakaus finds native marketplaces—like those for Codex or Claude Code—to be hit-or-miss. Specifically, he notes that the Claude Code update mechanism often fails due to caching issues. This is why he maintains his own CLI installer to ensure hooks are placed correctly within the system.

Why This Matters

The shift from prompting to harness engineering marks the professionalization of AI development. We are moving away from 'prompt alchemy'—where developers tweak words and hope for the best—toward actual software engineering for LLMs.

By treating the AI as one component of a larger system (including linters, scripts, and memory), we can create tools that are reliable and original. This prevents the 'homogenization' of the web, where every AI-assisted site looks identical because they all share the same latent space gravity.

Ultimately, this approach proves that the ceiling for AI quality isn't just about the model size; it's about the architecture of the harness surrounding the model. As Bakaus concludes, "If a gate can be skipped, it will be." Engineering the harness is the only way to ensure the model actually passes through it.

Frequently asked questions

What is 'AI Slop' in the context of design?

It refers to the generic, predictable aesthetic that LLMs tend to produce (like 'Claude beige') because they gravitate toward the most probable median of their training data.

Why does using scripts for instructions break prompt caching?

Prompt caching relies on the input being static. When a script injects dynamic data into the standard output, the input changes every time, making it impossible for the provider to cache the prompt.

What is the 'Inverted Judge' technique?

Because LLMs are often maximalists (thinking 'more is more'), Bakaus sometimes inverts the judge's score—assuming that if the AI rates a design very highly, it is likely over-decorated and lacks professional taste.

What is context pollution in LLMs?

It is the process of filling the model's context window with unnecessary metadata or redundant instructions, which can distract the model and lead to lower adherence to the primary task.

Sources

Source: “Beyond Prompting: Paul Bakaus on Harness Engineering” on YouTube by AI Engineer

Tags: LLM, Prompt Engineering, AI Design, Software Architecture, Harness Engineering, MCP

Related articles