返回资讯中心
外部精选
软件工程
#开发者实时流

What we've learned 3 years into harness engineering

What we’ve learned 3 years into Harness Engineering

Hacker News 最新bombastic3112026/10/20 · 26 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Agent Harness

Enterprise Ai

AI Agent

Llm Evaluation

What we’ve learned 3 years into Harness Engineering

Bassim Eledath

12 min read·

7 hours ago

--

Putting a great model in a bad harness is like putting Usain Bolt in flip-flops. We saw this a little while ago with GPT-5.6 Sol performing poorly on ARC-AGI-3 because of two mismanaged settings: retained reasoning and compaction. The default harness of the benchmark discarded the model’s reasoning after each move and dropped earlier actions as context filled up, so it kept starting over. Once both were enabled, the score nearly tripled from 13.3% to 38.3%.

At Asurion, a good harness is the difference between a customer getting the help they need and a conversation going in circles. We’ve been toiling with agentic frameworks since GPT-3.5, helping customers with problems as varied as a tablet that won’t charge, collecting the details a human expert needs to pick up a case, or activating a brand-new phone. This post traces how that thinking evolved over three years, culminating in AURA: a single harness we now use across many of our conversational AI use cases.

One of our early production use cases was tech troubleshooting over chat. A customer opens a session with something like “my phone won’t stop ringing,” and we walk them through a sequence of steps to diagnose and fix it. That sounds simple, and on the happy path, where the customer answers exactly as we expect at every turn, it is. But real sessions are rarely that straightforward. Customers often describe their problem loosely, so we first have to work out what they actually mean via clarifying questions (like “do you have an iPhone or an Android?” since each has different steps), and even after we have those preliminary details, they might switch intents midway through. For example, they might abandon the ringing problem three steps in to ask why their texts suddenly won’t send.

Decision trees choose the next troubleshooting step

Back when we didn’t have reasoning models, we had to rely on the harness to handle all of that messiness. Our first system, like many at the time, was built on LangChain and Haystack, and its intelligence was really two large hand-written decision trees. The first was a gatekeeper that ran on every message and asked a few things: should we even be troubleshooting right now? Is the customer asking for a human? Is this really a billing or account issue we can’t touch? Then we had a second tree that decided the actual troubleshooting move (or next best action): which device we’re on, which step comes next, and whether the customer had switched problems or was ready to give up and talk to a person. We also had several binary classifier LLMs that we called checkers, which fed those trees the yes-or-no facts they branched on, each answering one narrow question like “is the issue resolved?” or “are they ready for the next step?”

It was a lot. It gave us predictable behavior and kept the model on a short leash, but it was also an incredibly large codebase that was becoming overbearing to manage. Worse, it was rigid and very difficult to adapt to new use cases, particularly the one that had just popped up: guiding a caller through a structured information-collection flow over the phone, gathering the details a specialist needs to pick up their case (now an important use case for us).

So we built a second harness from scratch, this one specifically focused on voice. We did a few things differently this time:

A state machine guides each phase of the call

Latency optimizations: Waiting 5 seconds for a response on chat is okay (expected, even) but over a phone call it’s unbearable. So we split the work in two, a pattern we call the gateway-response architecture: one model decides, the other model tunes the wording. Say the bot just asked “when did you first notice this problem?” and the caller says “yesterday.” The first model, the gateway (smarter but slower), doesn’t write the reply at all. It just emits structured output, something like {"answer": "past", "valid": true}, meaning the caller gave a valid answer and it’s a date in the past. Then the second model, the response model (faster), formats the already-chosen next question into natural wording. The gateway model stays fast because it emits a few tokens instead of sentences, and the response model is quick because formatting is lighter than reasoning about what to do. Meanwhile the safety checkers (is the caller frustrated? asking for a human?) are designed so they don't add latency to the turn. In our experiments, running two models sequentially like this not only improved response quality; it was also surprisingly faster than having a single model (with a larger prompt) do everything end-to-end.

One model decides, another words the response

This harness did its job well, but like the troubleshooter before it (and the many other harnesses we were maintaining), it was purpose-built and overfit for a single job. Before long, we were managing several harnesses at once, each specialized for its own purpose. As usage of our services scaled, so did the maintenance burdens. If our LLM gateway changed its request schema, all our client services across the harnesses would need to be updated. A lot of duplicated effort.

There was clearly a strong need for consolidation, and when reasoning models became fast enough for conversational AI, this finally became possible. When you can finally trust a model to reason, harness engineering becomes much simpler (by goal, not practice). In essence, your harness only has two responsibilities:

And you leave the model to reason about the rest. It is worth pausing here to admire the simplicity of that as the rest of the blog post addresses the implications of it.

This realization resulted in the framework we now use called AURA (Asurion Unified Real-time AI). It is one harness designed to power any conversational AI use case, across channels (chat and voice today).

At the heart of AURA is what we call a “contributor,” which is a plugin that exposes context and/or tools, and can be active or inactive. Some examples:

RAG? Also just a contributor! It exposes a search_kb tool, and when the LLM calls it, the contributor runs the search and injects the matching articles right back into context.

We have contributors for channels too. For example, when a session is live voice, a contributor switches on that adds system prompt instructions to keep responses short and conversational.

Press enter or click to view image in full size

Each contributor provides context and tools

Notably, many of our deployments have an escalation contributor that is always active because a user may ask for a human agent at any time, and that exposes the necessary context and tools for the LLM to route to a human.

Under the hood, a contributor is just an object that answers three questions about the current state:

Here is a grossly simplified version of what an intent detection contributor looks like:

class IntentContributor(Contributor): def is_active(self, state, user_input) -> bool: return state.intent is None # active until an intent is on state

def get_injections(self, state, user_input) -> list[ContextInjection]: return [ContextInjection("intents", f"Figure out why the customer is reaching out. Options:{INTENTS}.")]

def get_tools(self, state) -> list[ToolDef]: return [record_intent] # records the intent to state

Why can’t the contributors be active all the time? The obvious answer here is noise. Every extra instruction and tool you hand the model is one more thing it can misread or misfire, and a longer prompt is slower and more expensive on every single turn for context the model was never going to use.

A natural follow-up question may be “why not have some semantic matching for contributors similar to how Claude Code has tool search?” Well, besides keeping context lean, contributors are the way to have some form of steerability. For example, if an action isn’t part of a given use case, we simply don’t load the contributor that provides it, so the model never has that tool to call, however the customer phrases the request. (We still instruct the model not to promise outcomes it can’t deliver: removing a tool removes the action, not the temptation to talk about it.) So being able to disable contributors is as important as being able to enable them.

The big payoff with contributors being these modular units of context/actions is composability. Any new client, for any use case, for any channel, is now just a list of contributors.

Different use cases compose different contributors

The rest of the harness

Beyond contributors, there are other primitives of AURA worth briefly mentioning:

Background reasoning leaves hints for the next turn

Observability

A harness engineering blog post wouldn’t be complete without us addressing observability in some way. We trace/record every turn of every conversation end to end through our observability platform, so any production session can be pulled back up, replayed, and inspected span by span, from the context each contributor injected down to every tool call the model made.

That tracing is also what powers what we call “expectations”, our umbrella term for the two forms of tests that guard the harness. Benchmarks are fixed expectations checked against a real turn or conversation. Some are blocking: verifiable, binary things that must always hold, and we treat a regression on any of them as a release blocker. For example, when a caller asks for a human, the bot must raise the escalate flag. If that does not hold true we cannot push to prod.

async def test_agent_request_escalates(): outcome = await run_turn( TurnInput(session_id="t-1", client_id="example_client", user_message="just get me a human"), client=example_client, store=InMemoryStateStore(), ) assert any(a.type == "escalate" for a in outcome.actions)

Others are non-blocking: expectations we don’t pass every time given the current state of the model and architecture, but that we steadily push toward 100%. A customer says roughly when something happened and the LLM has to turn that into the right date, the kind of thing that’s correct most of the time but not always.

A Non-blocking benchmark

Simulations are our third type of test and they are like integration tests: an LLM plays the customer across a whole conversation (say, a customer who isn’t very technical setting up a new phone and getting stuck halfway through), and success is more subjective, but it’s an important qualitative gate before we ship. Simulations can still include verifiable expectations, like “somewhere in this conversation the bot must call record_intent.” Between the two, we cover the bases: the things that must not regress, and the messy end-to-end behavior we can only really judge by playing it out.

Stepping back, AURA is logically (as well as in code) separated into three layers: config, contributors, and core.

Config is the client layer, pure configuration that composes contributors into a specific product, owned by the teams closest to each client or use case.

Contributors are the reusable plugins we talked about.

Core is the engine underneath: the turn loop, state in an in-memory cache, tracing in our observability platform, the client for our internal multi-provider model gateway, etc.

Core lives in its own repository on purpose, because it turns out not to be specific to conversational AI at all (one of our internal tools, a slide-deck creator, runs on the same core with an entirely different set of contributors).

A surprising byproduct of this is how our roles have been redefined by this architecture. We used to spend our time writing long prompts and standing up whole harnesses; now the job is mostly writing contributors. When a new client, use case, or channel shows up, the first question is no longer “what pipeline do we build?” but “what does the model need

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →