Your AI Keeps Giving Wrong Answers. More Data Might Make It Worse.

The same wrong answer can have three different causes, and each one needs a different fix. 

Here's a hypothetical. A customer support assistant tells a support rep a customer isn't eligible for a refund. The customer is. The answer sounded confident and was wrong.

A common response is to give the AI system more: another data source, more retrieved documents, a bigger context window, a chat history that never resets. Sometimes that's the right call. Other times it's how a team spends a quarter building retrieval infrastructure for a problem that was never a shortage of information.

In this refund case, there are at least three ways the assistant could have gotten it wrong, and the answer looks the same in each.

What went wrong?	In our example case:	What fixes it:
Missing evidence	The assistant never retrieved the customer's contract, so it applied the standard policy.	Supply the authoritative source.
Missing meaning	It retrieved the contract, but didn't know that the business treats certain mid-term downgrades as cancellations for convenience. That rule wasn't documented in any context the system could access.	Establish the definitions, rules, and relationships the system was never given.
Bad context selection	It had everything, but a stale policy sat next to the current one, and it cited the stale one.	Decide which source governs, and reconcile or remove the rest.

These can overlap. A stale policy and an undefined term can show up in the same wrong answer, so treat the table as a set of questions, not a sorting bin. A wrong recommendation doesn't tell you which cause you're dealing with, and adding more data only addresses the first. 

Evidence and meaning

Missing evidence and missing meaning both come down to what the AI system has to work with.

Evidence is the source material itself: the contract, the system of record. Hand the AI assistant the contract and it can find the clause and cite it. Withhold it, and unless the system is built to stop there, the AI system may produce a plausible refund policy instead of this customer's.

Meaning is what the contract doesn't say on its own. The contract might contain an exception for "cancellation for convenience," but the business may have established additional rules about which customer actions qualify, or how that exception interacts with other policies. Those rules may not appear anywhere in the contract itself.

That kind of knowledge often lives in a semantic layer, a business glossary, or an ontology, which is a structured representation of business concepts and how they relate to each other. Sometimes it still lives only in the heads of people who've run the business for years. Unless that knowledge is made available to the AI system, having the contract isn't enough.

Bad context selection 

In the third case, the assistant can reach the right policy, but it's sitting next to an older version, a duplicate export, and a draft nobody approved. Nothing is missing, so the problem doesn't look like missing information.

Some context platforms can scan across your systems and surface that two policies disagree. They can't decide which one governs. That's a business decision. Someone has to make it, own it, and keep it current when the policy changes.

Sometimes the fix is subtraction. Other times it's reconciliation: keeping both sources and making clear which one wins, and when.

The part that has to live outside the agent 

Asking the AI application to be more careful won't fix any of these. It can't check its answer against a contract it was never shown, and it has no reliable way to tell whether "not enough" or "too much" is why the answer came out wrong.

That work has to live in the system around the agent: which source is authoritative, what definitions apply, what's in scope for this decision, who owns the judgment, and whether the assistant is allowed to say nothing when the evidence it needs isn't there. Those are system-design decisions that no prompt can make for you.

A shared enterprise context layer can be a good foundation, and reusing one across applications makes sense. The risk is starting there, or pointing every AI application at everything it holds. Start with the decision the refund assistant supports and deliver the context that decision needs. Reuse the shared foundation as more use cases come online.

Three questions before you add anything

Summarizing a document someone just pasted in is mostly an evidence problem. These questions matter once an AI system answers business questions on its own, from more than one source, in front of people who act on the answer. When the refund recommendation comes back wrong, ask:

• Can you trace the recommendation to the customer's actual contract? If not, start with missing evidence.

• Is it applying the definitions the business has actually agreed on, not a plausible-sounding version of them? If not, suspect missing meaning.

• Is the policy it cited current, uncontested, and relevant to this customer? If it's stale, duplicated, or fighting with another source, adding more can make things worse instead of better.

What it takes to fix this

Fixing it starts with the decision:

1. Understand the decision. What is the assistant helping someone decide: approve, escalate, or decline?

2. Reconcile definitions and authority. Which policy governs, what "cancellation" means here, and who says so.

3. Establish ownership. Someone responsible for keeping those answers current as the business changes.

4. Design how the context is delivered. Retrieval, tools, and access, scoped to this use case, including when the assistant should escalate instead of answering.

5. Test that the system uses it correctly. Check that the answer traces to the right source and the right definition. A plausible answer proves nothing.

Retrieval comes last, because it follows from those choices.

That work rarely sits in one team's job description. Business owners understand the policies and decisions. Data teams understand the sources, definitions, and inconsistencies. AI architects understand how to deliver the right context and build systems that use it reliably.

The hard part is bringing those groups together to resolve the disagreements. Solving it often takes more than the team that built the application can do alone.