Building RAG Systems That Resolve Contradictory Sources Without Hallucinating
Hook: The Problem Nobody Talks About
You've built a RAG system. It retrieves documents. It generates answers. It works great... until your knowledge base contains sources that directly contradict each other.
Then things get weird.
Your AI might confidently tell a customer that Product X costs $99 while simultaneously saying it costs $149. Or it retrieves research papers that fundamentally disagree about a scientific finding. Or worse—it invents a third answer that combines the worst of both contradictions.
This isn't a rare edge case. It's the default state of most real-world knowledge bases. Contradictions hide everywhere: outdated information mixed with current data, different perspectives on the same topic, conflicting expert opinions, or simple human error across documents created by different teams at different times.
The question isn't "Will your RAG system encounter contradictions?" It's "When it does, will it handle them gracefully—or will it hallucinate?"
Let me show you how to build systems that actually deal with this.
What You Will Learn
By the end of this post, you'll understand:
This isn't theoretical. We'll work through actual patterns used in production systems handling millions of queries.
Simple Explanation: The Restaurant Menu Analogy
Imagine you're building a chatbot that answers questions about a restaurant.
Your knowledge base has two documents:
A basic RAG system might retrieve both documents and feed them to an LLM with a prompt like: "Using these sources, answer: How much does the burger cost?"
Now the LLM is confused. It sees conflicting information. Under pressure to be helpful, it might:
All of these are hallucinations—the model inventing information to resolve cognitive dissonance.
A better system would:
That's the difference between a RAG system that hallucinates and one that reasons carefully.
How It Works: Three Detection Patterns
Pattern 1: Semantic Similarity + Assertion Extraction
The first step is detecting when you've actually retrieved conflicting information—not just different phrasings of the same fact.
python
Pseudo-code for contradiction detection
def extract_assertions(text):
"""
Break text into atomic claims that can be compared.
"The product costs $99 and ships in 2 days" becomes:
- PRICE: 99
- SHIPPING_TIME: 2_days
"""
# This can use zero-shot prompting with Claude/GPT-4
# or trained NER models for your domain
pass
def detect_contradiction(assertion_1, assertion_2):
"""
Compare assertions for direct conflict.
Returns: (is_contradictory, confidence, conflict_type)
"""
# Same attribute + different values = contradiction
if assertion_1.attribute == assertion_2.attribute:
if not values_are_compatible(assertion_1.value, assertion_2.value):
return True, confidence_score, "direct_contradiction"
return False, 0, None
The key insight: You need to decompose sources into claims before comparing them. "Expensive" contradicts "affordable". But "expensive" doesn't contradict "ships fast"—those are different attributes.
Pattern 2: Source Credibility Scoring
Once you detect a contradiction, you need to decide which source wins.
Don't default to recency (newest isn't always best). Instead, build a credibility score:
python
def score_source_credibility(source, context):
score = 0
# Recency (but not too much)
days_old = (today - source.date).days
if days_old < 30:
score += 3
elif days_old < 90:
score += 2
elif days_old < 365:
score += 1
# Author credibility (if available)
if source.author_is_domain_expert:
score += 5
if source.author_is_internal_team:
score += 2
# Specificity and detail
if source.word_count > 500:
score += 2
if source.has_citations:
score += 3
# User ratings (if available)
if source.helpfulness_rating:
score += source.rating * 2
return score
This scoring system can be tuned for your domain. A medical AI might weight "comes from peer-reviewed journal" much higher. A product support bot might weight "written by current product manager" highest.
Pattern 3: Explicit Contradiction Handling in the Prompt
Once you've detected and scored sources, tell your LLM about it explicitly.
python
def build_contradiction_aware_prompt(retrieved_sources, detected_contradictions):
prompt = """You are answering a question using the provided sources.
IMPORTANT: You have retrieved sources that CONTRADICT each other.
Here is the conflict:
"""
for contradiction in detected_contradictions:
source_a = contradiction['source_1']
source_b = contradiction['source_2']
credibility_a = contradiction['credibility_score_1']
credibility_b = contradiction['credibility_score_2']
prompt += f"""\n
CONTRADICTION DETECTED:
Prefer Source {'A' if credibility_a > credibility_b else 'B'} due to higher credibility score.
"""
prompt += """\n\nYour task:
Never invent a middle ground or third answer."""
return prompt
This tells the LLM: "You have a conflict. Here's how to think about it. Don't hallucinate." Much more effective than hoping the model figures it out.
Real World Example: Building a Product Support RAG
Let's build this for a real scenario. You're supporting a SaaS product with a knowledge base of:
A customer asks: "Does your API support OAuth 2.0?"
Your system retrieves:
Without contradiction handling, the LLM might respond: "OAuth 2.0 will be coming soon and is fully supported," which makes no sense.
With contradiction handling:
python
Step 1: Extract assertions
source_1_assertions = [
{"attribute": "oauth2_support", "value": "full", "source": "2024_docs"}
]
source_2_assertions = [
{"attribute": "oauth2_support", "value": "not_yet", "source": "2022_wiki"}
]
source_3_assertions = [
{"attribute": "oauth2_support", "value": "in_progress", "source": "2023_blog"}
]
Step 2: Detect contradictions
contradictions = [
{"assertion_1": source_1, "assertion_2": source_2, "type": "direct"},
{"assertion_1": source_1, "assertion_2": source_3, "type": "direct"},
]
Step 3: Score credibility
credibility = {
"2024_docs": 9, # Official, recent
"2022_wiki": 2, # Old, unofficial
"2023_blog": 5, # Semi-recent, public
}
Step 4: Winner is clear
preferred_source = "2024_docs"
Step 5: Generate response
response = """
Yes, OAuth 2.0 is fully supported in our API as of 2024.
Note: This was added after 2022. If you're reading old documentation,
please refer to our current API docs at [link].
"""
The user gets a clear, confident answer with context about why older information existed. No hallucination. No middle ground that doesn't exist.
Why It Matters in 2026
Here's what's changing:
Larger knowledge bases: Companies are ingesting more documents into RAG systems. More documents = more contradictions statistically.
Multi-source retrieval: By 2026, systems won't just pull from internal docs. They'll blend internal data with real-time APIs, customer feedback, and external sources. More sources = more conflicts.
Regulatory pressure: If your AI makes contradictory claims in regulated industries (healthcare, finance, legal), you're liable. "The AI hallucinated because sources conflicted" won't hold up in court.
User expectations: Early RAG users forgave weird outputs. 2026 users won't. They expect intelligent reasoning, not confident nonsense.
Competitive advantage: The RAG systems that handle contradictions well will win. Users will prefer systems that say "Here's what we know and why" over systems that confidently contradict themselves.
Common Misconceptions
Misconception 1: "Just use the newest source."
Wrong. Sometimes old information is more carefully written. Sometimes new documents are quick patches with typos. Sometimes the newest source contradicts the truth because it's based on incomplete data. Score credibility holistically.
Misconception 2: "The LLM will figure it out."
Nope. LLMs are trained on internet text where conflicting information *is* common. They've learned to synthesize, average, or hallucinate rather than flag conflicts. You have to tell them explicitly.
Misconception 3: "Just remove contradictory documents."
Impractical. You can't manually review thousands of documents. Also, sometimes you *want* to show both perspectives (different pricing for different regions, for example).
Misconception 4: "This is too complicated for my RAG system."
It's not. Once you add assertion extraction and credibility scoring, most contradictions resolve automatically. You're adding maybe 5-10 lines of logic.
Misconception 5: "Contradiction resolution = always pick one source."
Sometimes you should present both. "We offer two pricing tiers: $99/month (standard) or $199/month (enterprise). Previously we only offered the standard tier." That's contradiction resolution that provides value.
Key Takeaways
What To Do Next
This week: Audit your RAG system's top 10 queries. Manually check if retrieved sources ever contradict. Estimate how often it happens.
Next week: Implement basic assertion extraction. Use an LLM to break sources into claims. Don't overthink it—a simple prompt works.
Then: Build credibility scoring tuned to your domain. What signals matter most to your users?
Finally: Update your system prompt to explicitly handle detected contradictions.
Start small. Maybe 20% of queries involve contradictions. As you solve those, your system gets smarter and more trustworthy.
The RAG systems that matter in 2026 won't be the ones that retrieve documents fastest. They'll be the ones that reason about what they retrieved—especially when the sources disagree.
Now go build something that doesn't hallucinate when reality is messy.