Back to Blogs
Real Explainability or Marketing? How to Test a Vendor’s XAI Claims Before You Buy
Explainable AI tools produce reasons for an AI system’s output: which inputs influenced the output or which documents the answer drew on. A vendor that calls its product explainable is claiming those reasons are true. Before you sign, you can test that claim by running three checks on the vendor’s system using your own data.
This kind of testing is most important if your team buys AI for mission-critical and compliance decisions across government and commercial use cases such as insurance claims, risk, compliance, and model evaluation, to name a few. It’s important to know what’s under the hood as decisions made with XAI software will need to be explained, long after the contract is signed.
What real explainability means
We consider an explanation real when it meets three conditions. It traces to the data, model, and steps that actually produced the output. The person who has to act on it can understand and check it. And it stays consistent: the same output gets the same explanation every time. An explanation that reads well but fails the first condition does the most harm, because it gives users confidence the facts don’t support.
NIST’s four principles of explainable AI draw the same line and treat explanation accuracy as separate from decision accuracy. Explanations shouldn’t just stay inside the team that built them: when an auditor challenges a decision, the explanation is your evidence.
Three tests that turn a claim into evidence
1. Reproduce it
Ask the vendor to run the same input twice, then again with a trivial change such as reordered fields. If the explanation changes substantially while the output stays the same, it is describing noise rather than the model.
2. Change the cause
Take the factor the explanation says mattered most and change or remove it. If a generative system cites a document, remove it and check whether the answer changes. When the output barely moves, the explanation is just a decoration.
3. Repeat both on your own cases
A method that is faithful on one dataset can fail on another. Run both tests on a sample of your own records during a proof of concept, and record each test, its inputs, and the result.
Why a convincing explanation can still be wrong
Every common type of explanation has documented failure modes. Attribution methods such as SHAP and LIME often disagree and can be manipulated to hide bias. A 2023 audit of four generative search engines found that only 74.5% of citations supported the claims they accompanied.
Written reasoning has the same problem. In 2025, Anthropic slipped a hint about the answer to a test question into a model’s input and checked whether its reasoning admitted using that hint. Claude 3.7 Sonnet mentioned it in 25% of cases and DeepSeek R1 in 39%, in scenarios Anthropic called somewhat contrived. You can perform the same check by planting a hint of your own.
What regulators ask for
None of these frameworks mandates a specific technique. The EU AI Act gives people affected by most Annex III systems a right to clear and meaningful explanations, with those rules applying from December 2, 2027, and Regulation B requires the specific principal reasons for adverse action. SR 26-2 leaves generative AI to a bank’s own governance; see our guide to evaluating an AI governance platform under SR 26-2.
Questions to ask a vendor
In the RFP, it’s critical to verify that the vendor can show evidence for each answer, not just describe a feature.
- Which explanation methods does the product use, and for which kinds of models?
- Do explanations change when the factors they name are removed?
- Does a citation mean the model used that source, or only that it is related?
- Can we run a proof of concept on our own data and keep the records?
Ask for the model card too, but read it as a starting point rather than evidence. As our colleague Ben Faircloth has argued, public benchmarks show whether a model is generally capable, not whether it performs on your data and your task, and the same limit applies to a model card. In many cases a smaller model would better serve the task at hand, and a transparent model leaves no separate explanation layer to test.
Test explainability claims on your own data
We built SeekrGuard™ to independently assess AI models and agents in high-stakes environments, using your own data and evaluating them against your criteria. For generated answers, SeekrFlow™ includes context attribution, which tests how a response changes when sources are removed. See also our enterprise guide to explainable AI.
Book a consultation with an AI expert and our team will walk you through a live demo.