Back to Blogs
Agents Need to Explain Themselves While They Work
NVIDIA recently introduced its Open Agent Safety Platform, and with it a clear argument: controls on AI agents belong close to the hardware, out of the agent’s reach. Put simply, the more an agent can do, the more its reasoning needs to be visible. We agree. And we’d add a question the industry now has to answer: visible to whom, and how fast?
The math of long-running agents
Consider an agent that makes the right decision 99% of the time at every step. That sounds excellent. Now give it a task with many steps. Following the logic of compounding probability, after 10 steps, the chance that every step went right drops to about 90%. By roughly 70 steps, it’s a coin toss. After 100, it’s about 37%.
Real tasks are messier than that illustrative calculation, but the direction holds. Agents are moving rapidly toward these long, multi-step tasks: working across systems, calling tools, revising their own plans. Errors rarely arrive as one dramatic failure, but they accumulate and one wrong answer or assumption at any given step, especially early on, shapes everything after it.
Why after-the-fact review falls short
Most attempts at explainability – the ability to understand why a model or agent acted a certain way – are really just observability after the fact. Most approaches to explainability are really just observability after the fact.
The agent acts, an output appears, and then a human looks through traces to find why the outcome appeared. That can work when a reviewer knows what the right answer looks like and has time to investigate. But in regulated or mission-critical environments, “we’ll figure out why afterward” isn’t explainability – it’s too late.
For long-running agents, that requirement for time and expertise further complicate the issue because no one can review every step of every agent and many organizations deploy AI precisely where no one already has the answer.
So explanation has to move inside the agent’s workflow. At each consequential step, the agent should explain the decision it’s about to make, validate it against the evidence, and correct course before an error propagates. The process should be proactive, automated, and intrinsic to the agent’s reasoning, allowing it to recover from errors throughout its trajectory and prevent a single mistake from cascading into failure – not a person watching a screen and reacting after the fact.
The harness is now part of the explanation
Explainability research has mostly focused on the model itself: which training data shaped a behavior, which part of the input drove an output, what’s happening in the model’s internal activity. Those questions remain essential.
But models now run inside agent harnesses, the software that manages an agent’s plan, tools, memory, and context. The harness makes decisions that shape the outcome: which path to take, which tool to call, what to add to or drop from context.
That makes the harness a source of explanation in its own right. When an agent goes wrong at step 40, the model’s reasoning at step 40 may be perfectly sound. The problem may be that a tool returned bad data at step 22, or that a constraint from step 3 was trimmed from context to make room. The explanation lives in what the harness did along the way, not only in what the model concluded at the end.
Explanation at the speed of the agent
There’s a practical obstacle. Attribution is computationally expensive. Methods that work well for reviewing a single answer become impractical when they have to run at every step of a long task, across many agents at once.
That’s why our research team is implementing attribution methods as efficient GPU kernels that run inside the agent harness, so explanation keeps pace with the agent instead of trailing behind it. If safety controls are moving closer to the chip, explanation should move there too.
Speed is only part of the problem. The harness also needs to determine when a decision calls for deeper verification and what to do with the result. Attribution can show which evidence influenced an output, but that evidence may still be wrong or incomplete. Turning that insight into reliable decisions about when to proceed, seek more evidence, or stop remains an open research problem.
This complements containment rather than competing with it. Containment decides what an agent is allowed to do and stops it when it goes too far. Explanation shows why the agent acted and whether its reasoning holds up. An organization with both can give agents more authority with confidence. An organization with only containment has to choose between a short leash and a leap of faith.
Why openness matters
NVIDIA noted that one advantage of open models is that their reasoning and internal activity are visible. That’s a practical point, not just a philosophical one.
You can only explain what you can see, and deep explanation depends on access to a model’s internals – what actually drove the output, not just a description of it after actions were already taken. The most reliable explanation methods measure what happens inside a model as it works, and closed models accessed through an API show only what goes in and what comes out. Open-weight models show the rest.
The visibility and explainability needed also depend on an open ecosystem, where explanation can run wherever agents run, on whatever platforms an organization has chosen. That’s the ecosystem we intend to be part of.
What’s still unsolved
Much of this is early. The kinds of evidence available keep expanding, and which evidence matters most depends on the domain and the application. Turning explanations into reliable automatic corrections without introducing new errors is still an open research problem. We’ll publish what we learn, including where it doesn’t work.
The direction is clear, though. Organizations can give more authority to agents that explain and check their own decisions as they work. That’s the standard we’re working toward.
