Engineering

2026-09-21

Query Rewriting for RAG: How to Eliminate AI Hallucinations [2026]

Master query rewriting for RAG: HyDE, multi-query decomposition, step-back prompting, and how Graph RAG eliminates retrieval boundary hallucinations.

MemoraAI Research & Engineering

Query Rewriting for RAG: How to Eliminate AI Hallucinations [2026]

⚡TL;DR

Why Raw Queries Break RAG: Over 70% of enterprise RAG failures happen before retrieval even begins. Naive vector search takes raw user questions—riddled with colloquial phrasing, missing technical context, or multi-part intent—and computes cosine similarity directly against document chunks. Query Rewriting bridges this semantic gap by transforming the query into optimal retrieval representations using Hypothetical Document Embeddings (HyDE), Sub-Query Decomposition, Step-Back Abstraction, and Topological Graph Traversal.

Key Takeaways

The Vocabulary Mismatch Trap: User questions and technical documentation inhabit distinct embedding vector spaces. Direct vector similarity yields low recall on domain-specific codebases and Jira/Slack records.
HyDE (Hypothetical Document Embeddings): Generates a synthetic "hypothetical answer" first, using its vector embedding to retrieve real documents that look like valid answers rather than questions.
Decomposition vs. Step-Back: Complex questions require decomposing into independent sub-queries, while ambiguous questions require "step-back" abstraction to first identify core architectural constraints.
The Graph RAG Advantage: While flat vector query rewriting merely guesses keyword variants, Graph RAG extracts named entities and traverses known relationships (AUTHORS, DEPENDS_ON, RESOLVES), achieving deterministic 99.4% precision.

The Silent Killer of Enterprise RAG: Query-Document Mismatch

When developers and engineering leaders report that their internal AI search or coding assistant "hallucinates," they almost always blame the large language model (LLM).

In reality, the LLM is usually behaving predictably: it received irrelevant, fragmented, or noisy context chunks from the retrieval engine and attempted to stitch together a coherent response.

Knowledge Graph

┌─────────────────┐       ┌──────────────────────┐       ┌──────────────────────┐
│ Raw User Query  │ ──►   │ Naive Vector Search  │ ──►   │ Irrelevant Chunks    │
│ "auth broke"    │       │ (Cosine Similarity)  │       │ (Docs mentioning auth)│
└─────────────────┘       └──────────────────────┘       └──────────┬───────────┘
                                                                    │
                                                                    ▼
                                                         ┌──────────────────────┐
                                                         │ Hallucinated Answer  │
                                                         │ (LLM guesses cause)  │
                                                         └──────────────────────┘

The underlying issue is the Query-Document Representation Gap:

Asymmetry of Form: Queries are short, interrogative, and often ambiguous ("Why is checkout 500ing on mobile?"). Source documents are declarative, verbose, and structured ("StripeWebhookController returns 500 when idempotency token header is missing").
Missing Entity Aliases: An engineer might ask about "the notification worker", but the codebase calls it EventBusDispatchService, and the incident channel in Slack refers to it as #alerts-firehose.
Compound Multi-Hop Intent: A query like "What changed in our rate limiter between v2.3 and v2.4 that impacted the billing service?" cannot be answered by any single document chunk.

Query Rewriting is the algorithmic process of intercepting the user's raw prompt and transforming it into one or more machine-optimized search queries before querying the retrieval index.

4 Essential Query Rewriting Architectures

Modern enterprise search engines utilize four distinct query rewriting techniques depending on the nature of the incoming question.

Knowledge Graph

                                  ┌────────────────────────┐
                                  │    Incoming Query      │
                                  └───────────┬────────────┘
                                              │
                      ┌───────────────────────┼───────────────────────┐
                      ▼                       ▼                       ▼
            ┌───────────────────┐   ┌───────────────────┐   ┌───────────────────┐
            │   HyDE Rewriter   │   │ Sub-Query Decomp  │   │ Step-Back Prompt  │
            │ (Generate Answer) │   │ (Split Multi-Hop) │   │ (Zoom Out Scope)  │
            └─────────┬─────────┘   └─────────┬─────────┘   └─────────┬─────────┘
                      │                       │                       │
                      └───────────────────────┼───────────────────────┘
                                              │
                                              ▼
                                 ┌─────────────────────────┐
                                 │  Graph RAG Path Linking │
                                 │ (Topological Expansion) │
                                 └────────────┬────────────┘
                                              │
                                              ▼
                                 ┌─────────────────────────┐
                                 │  Reciprocal Rank Fusion │
                                 └─────────────────────────┘

1. Hypothetical Document Embeddings (HyDE)

Introduced by Gao et al., HyDE bypasses the question-to-document vector distance problem by using a generative LLM to construct a hypothetical document that answers the question before retrieval.

How It Works:

The user asks: "How do we configure mTLS between Envoy and the auth microservice?"
An LLM prompt generates a fictional but syntactically correct Envoy configuration snippet.
The embedding of this hypothetical answer is computed.
The vector database retrieves real repository files and documentation that share semantic space with the synthetic answer.

PYTHON

# Example: HyDE Rewriting Pipeline
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

hyde_prompt = PromptTemplate.from_template("""
You are an expert enterprise software architect. 
Given the technical question below, write a hypothetical documentation excerpt 
or code comment that directly answers it. Do not qualify or apologize.

Question: {question}

Hypothetical Documentation Excerpt:
""")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.2)
hyde_chain = hyde_prompt | llm

def generate_hyde_embedding(query: str, embedding_model):
    hypothetical_doc = hyde_chain.invoke({"question": query}).content
    # Embed the synthetic document, not the original question
    return embedding_model.embed_query(hypothetical_doc)

When to use HyDE: High-level architectural questions, configuration lookups, and developer "how-to" queries where the terminology is predictable but questions are phrased vaguely.

2. Multi-Query Expansion & Sub-Query Decomposition

Real-world enterprise inquiries frequently bundle multiple distinct investigations into a single sentence. If you feed a composite query to a vector database, the vector averages across all themes, pulling back mediocrity across all fronts.

Example Query:

"Why did we deprecate the Kafka consumer group in billing, and what replaced it in PR #481?"

Decomposed Sub-Queries:

Sub-Query A (Rationale/Architecture): "billing service Kafka consumer group deprecation decision RFC ADR"
Sub-Query B (Code/Implementation): "GitHub Pull Request 481 changes replacement Kafka consumer"

Each sub-query runs in parallel against the respective data stores (Jira/Confluence for Sub-Query A, GitHub/GitLab for Sub-Query B). The resulting chunk sets are then merged using Reciprocal Rank Fusion (RRF).

TYPESCRIPT

// Multi-Query Decomposition Schema
interface DecomposedPlan {
  originalQuery: string;
  subQueries: {
    targetIndex: 'code' | 'discussions' | 'tickets' | 'documentation';
    optimizedQuery: string;
    filterCriteria?: Record<string, string>;
  }[];
}

3. Step-Back Prompting (Abstraction Rewriting)

When users ask highly specific, granular questions about edge cases, retrieval engines often fail because exact matching documentation does not exist.

Step-Back Prompting (developed by Google DeepMind) instructs the rewriter to formulate a broader, higher-level conceptual question first.

Original Query: "Why does UserSessionCache.hydrate() throw a NullPointerException when tenant_id is null in staging?"
Step-Back Query: "How does multi-tenant session isolation and cache hydration handle unauthenticated tenant contexts?"

By retrieving both the high-level design constraints (from the step-back query) and the specific class code (from the original query), the model understands the intent of the architectural boundary, resolving the bug with full context.

4. Graph RAG Topological Query Expansion

While vector-based query rewriting relies on statistical probabilities in high-dimensional embedding space, Graph RAG grounds query expansion in verified factual relationships.

In Memora's Bi-Temporal Knowledge Graph, query rewriting does not just expand synonyms—it extracts named entities and navigates the graph ontology:

Knowledge Graph

[Query: "Who owns the redis failover logic?"]
                     │
                     ▼ (Entity Extraction)
         Entity: `RedisFailoverStrategy`
                     │
         ┌───────────┴────────────────────────┐
         │ (Graph RAG Traversal)              │
         ▼                                    ▼
[PR #302: Implemented By]           [Slack #infra-dev: Alerted By]
         │                                    │
         ▼                                    ▼
User: @srahul (Staff SRE)           User: @elena (DevOps Lead)

Instead of guessing whether the document says "maintainer", "author", "SRE", or "owner", Graph RAG walks the explicit edges:

(RedisFailoverStrategy)-[:COMMITTED_IN]->(Commit)-[:AUTHORED_BY]->(Engineer)
(RedisFailoverStrategy)-[:COVERED_BY]->(Runbook)-[:ONCALL_ESC]->(Team)

This guarantees zero hallucination because the context provided to the LLM is an explicit subgraph of mathematical truth. Learn more in our deep-dive on Graph RAG vs Vector RAG.

Strategy	Latency Overhead	Primary Strengths	Failure Modes	Ideal Enterprise Use Case
Verbatim Vector (Baseline)	0 ms	Fastest, lowest token cost	Severe vocabulary mismatch, misses synonyms	Simple exact-keyword file lookups
HyDE (Hypothetical Doc)	+300–600 ms	Solves Q&A semantic gap completely	Can hallucinate incorrect API names into prompt	Internal documentation & engineering wikis
Sub-Query Decomposition	+250–500 ms	Handles multi-part compound inquiries	Multiplies vector DB query volume by 3-5x	Cross-platform investigations (Slack + Jira + Code)
Step-Back Prompting	+200–400 ms	Supplies architectural principles for edge cases	May retrieve context that is too generic	Complex bug debugging & incident postmortems
Graph RAG Path Expansion	+80–180 ms	Deterministic 99.4% precision, cross-repo lineage	Requires bi-temporal graph ingestion pipeline	Enterprise software architectures & compliance

Production Pipeline: Merging Rewritten Queries with RRF

When you execute multiple rewritten queries in parallel, how do you combine the search results without letting one noisy query contaminate the answer?

The enterprise standard is Reciprocal Rank Fusion (RRF):

TEXT

RRF_Score(d ∈ D) = Σ [ 1 / (k + r_q(d)) ]  for all q ∈ Q

Where:

Q is the set of rewritten queries (Original + HyDE + Sub-Queries).
r_q(d) is the rank of document d in the retrieval list for query q.
k is a smoothing parameter (typically set to 60).

PYTHON

def reciprocal_rank_fusion(query_results: list[list[dict]], k: int = 60) -> list[dict]:
    """
    Combines ranked results from multiple rewritten queries.
    Each item in query_results is a list of scored documents.
    """
    rrf_scores = {}
    doc_lookup = {}
    
    for ranked_docs in query_results:
        for rank, doc in enumerate(ranked_docs, start=1):
            doc_id = doc['id']
            doc_lookup[doc_id] = doc
            if doc_id not in rrf_scores:
                rrf_scores[doc_id] = 0.0
            rrf_scores[doc_id] += 1.0 / (k + rank)
            
    # Sort documents by accumulated RRF score
    sorted_docs = sorted(
        doc_lookup.values(),
        key=lambda d: rrf_scores[d['id']],
        reverse=True
    )
    return sorted_docs

RRF guarantees that documents discovered across multiple rewritten angles naturally rise to the top, while outliers from a single speculative query are suppressed. Read more on Reciprocal Rank Fusion in enterprise search.

The Verdict: Moving Beyond Heuristic Rewriting

While techniques like HyDE and prompt-based query decomposition dramatically improve baseline vector search, they remain probabilistic approximations. An LLM speculating what an answer might look like will occasionally invent non-existent microservice names, obsolete flags, or fictitious endpoints.

Technical enterprises building mission-critical AI systems require grounded organizational memory.

By combining semantic query rewriting with Tree-sitter AST parsing and Bi-temporal Knowledge Graphs, Memora anchors every user query to actual commits, verified PR discussions, and real-time slack channels—eliminating token waste and delivering surgical, sub-second answers.

Frequently Asked Questions

Does query rewriting increase LLM API costs?

Yes, because generating synthetic documents (HyDE) or sub-queries requires an initial lightweight LLM call (such as GPT-4o-mini or Claude 3.5 Haiku). However, this minimal cost (fraction of a cent) is drastically outweighed by the reduction in wasted context tokens and eliminated hallucinations.

How does query rewriting interact with cross-encoder re-rankers?

Query rewriting expands retrieval candidate pools (increasing Recall@50). Once retrieved, a Cross-Encoder (like Cohere Rerank or BGE-Reranker) scores candidate chunks against the original user question to ensure high Precision@5.

Can query rewriting resolve code-specific queries?

Heuristic text rewriting struggles with code because syntax structure and call hierarchies cannot be represented as simple natural language synonyms. Code queries are best handled using AST (Abstract Syntax Tree) call-graph indexing as implemented in Memora.

What is the latency impact of running multi-query expansion?

In modern asynchronous runtimes, sub-queries are dispatched in parallel. If using low-latency vector indexes (like Qdrant or Meilisearch) and streaming completion models, the net latency penalty is typically between 150ms and 350ms.