A RAG application can look impressive during a product demo and still fail badly when real customers start asking questions.

Your team asks:

“What is our refund period?”

The system retrieves the correct policy and answers perfectly.

Everyone is happy.

Then a customer asks:

“I bought the annual Business plan six months ago. Can I cancel now, and what happens to the remaining payment?”

Suddenly the system retrieves an outdated policy, mixes it with unrelated subscription information and gives a confident answer that is only partly correct.

The chatbot did not completely fail.

That is what makes the problem dangerous.

It produced something believable.

This is why RAG evaluation before launch needs to test much more than whether the chatbot can produce fluent answers.

A production RAG system has two separate jobs:

1. Retrieve the right evidence.
2. Use that evidence correctly.

Microsoft’s current RAG evaluation framework separates these areas in the same way. It evaluates retrieval quality independently from response qualities such as groundedness, relevance and completeness.

If your team evaluates only the final answer, you may never discover where the real problem lives.

Start by Testing Retrieval Separately From Generation

Suppose a customer asks:

“Does the Enterprise plan include SAML SSO?”

Your chatbot answers:

“Yes, Enterprise customers can use SAML-based single sign-on.”

The answer is correct.

But imagine the retrieval system actually returned documentation about OAuth, password policies and a two-year-old pricing page.

The language model happened to know the correct answer.

Your test passes.

Your RAG system does not.

That matters because the same system could guess incorrectly on the next question.

For every important test case, inspect three things separately:

Query: What did the user ask?

Retrieved context: What evidence did the search layer find?

Final response: What did the language model say?

This simple separation makes RAG failures much easier to diagnose.

ZA Technologies’ existing article on why AI chatbots give weak answers explains the same principle: when a response fails, teams should ask whether the correct document was available, whether retrieval found it and whether the model actually used the retrieved evidence. Why AI Chatbots Give Weak Answers and How RAG Fixes Them

Build a Real Test Dataset Before Launch

Do not evaluate RAG by asking random questions during a meeting.

Create a repeatable evaluation dataset.

For each test query, record:

• The user question
• The expected source document
• The important facts that should appear
• Relevant document or chunk IDs
• An acceptable answer
• Information the system must not claim
• Whether the correct behavior is to refuse or escalate

Imagine you are testing an HR knowledge assistant.

A useful test case could look like this:

Question: How many paid personal days do Toronto employees receive during their first year?

Correct source: Canadian Employee Handbook, Personal Leave section.

Required fact: Three paid personal days.

Incorrect sources: US Employee Handbook and outdated 2024 handbook.

Expected behavior: Answer three days and cite the Canadian policy.

This gives you something far more useful than saying:

“The answer sounds good.”

Test Whether the Retriever Finds the Right Documents

Retrieval quality determines what evidence the model sees.

If the search layer retrieves poor evidence, the generation layer starts with a disadvantage.

Microsoft’s current RAG evaluators include document retrieval metrics designed to measure whether known relevant documents are being returned and whether the strongest results appear near the top of the ranking. Its evaluation framework includes measures such as NDCG, Fidelity and maximum relevance in the top retrieved results.

You do not need to turn every product manager into an information retrieval researcher.

For practical SaaS testing, start with a few simple questions.

Did the correct document appear?

For each query, check whether the source containing the answer was retrieved.

How high did it rank?

The right document appearing at position 15 is much less useful if your application sends only the top five chunks to the language model.

Did irrelevant documents dominate the results?

A RAG system can technically retrieve the right source while surrounding it with large amounts of noise.

Are old versions outranking current documents?

This is particularly dangerous for:

• Pricing
• Policies
• Product documentation
• Compliance rules
• Technical releases

Retrieval testing should therefore care about authority and freshness, not just semantic similarity.

Do Not Test Only Perfectly Worded Questions

Internal teams often evaluate AI using questions written by the people who built the knowledge base.

Those questions tend to match the documentation language.

Real customers do not speak like your documentation.

Your help center may say:

“Subscription termination is non-refundable after the initial 14-day period.”

A customer asks:

“If I leave after two months, do I get my money back?”

Your evaluation set should include both.

Test:

• Exact terminology
• Plain English
• Misspellings
• Abbreviations
• Long questions
• Short questions
• Vague questions
• Follow-up questions

A good retrieval system should handle different ways of expressing the same need.

Groundedness: Did the Model Stay Inside the Evidence?

Getting the correct document is only half the job.

Now ask:

Did the model actually stick to what the document says?

Microsoft defines groundedness as the degree to which the generated answer aligns with the supplied context without inventing unsupported information. It distinguishes this from completeness, which asks whether important expected information was omitted.

Consider this retrieved policy:

“Business plan customers may cancel at any time. Payments already made are non-refundable.”

Good answer:

“You may cancel the Business plan at any time, but payments already made are non-refundable.”

Weak answer:

“You may cancel at any time and will usually keep access until the end of your billing period.”

Where did the second claim come from?

It may sound reasonable.

The retrieved evidence did not say it.

That is a groundedness failure.

For business AI systems, the difference between plausible and supported information is critical.

Completeness Matters Too

A response can be perfectly grounded and still be poor.

Suppose the source says:

“Enterprise customers receive 24/7 support by phone and email. Critical incidents have a one-hour initial response target.”

The chatbot answers:

“Enterprise customers receive 24/7 support.”

Everything it said is supported.

But it missed the one-hour response target, which may have been the reason the user asked.

This is why evaluation should include response completeness.

Microsoft describes groundedness as similar to precision, where unsupported content is avoided, while completeness focuses more on recall, meaning important expected information is not missed.

A strong answer needs both.

Test Questions With No Valid Answer

This is one of the most valuable RAG tests and one of the most frequently skipped.

Ask questions that your knowledge base cannot answer.

For example:

“Will your company open an office in Singapore next year?”

If no approved source contains that information, what should happen?

A dangerous chatbot might answer:

“Yes, the company plans to expand into Singapore next year.”

A safer system should say something such as:

“I don’t have verified information about future office expansion.”

RAG evaluation should explicitly measure how the system behaves when retrieval produces weak or irrelevant evidence.

Your chatbot needs to know when it does not know.

That capability often builds more customer trust than producing an answer to every question.

Include Conflicting Documents

Real business knowledge bases are rarely clean.

You may have:

• An old pricing PDF
• A new website pricing page
• An internal policy draft
• A final approved policy
• Duplicate help center articles

Create tests where conflicting information exists.

Then ask:

• Which source wins?
• Does the retriever understand version metadata?
• Is the newest document always appropriate?
• Does an approved policy outrank a draft?
• Can obsolete documents be excluded completely?

This often exposes knowledge-management problems that prompt engineering cannot solve.

Google’s RAG guidance notes that evaluation can help teams improve source data, parsing, chunking, search configuration and query handling, which reinforces an important lesson: RAG quality is a pipeline problem, not simply a model problem.

Evaluate Chunking, Not Just the Model

Your source document can be correct while your retrieved chunk is incomplete.

Imagine the original document says:

Paragraph 1: Premium accounts include international transfers.

Paragraph 2: International transfers are limited to $20,000 per transaction.

If those paragraphs are separated badly during chunking, a question about transfer limits may retrieve only the first paragraph.

The model cannot provide information it never received.

When retrieval fails, experiment with:

• Chunk size
• Chunk overlap
• Heading-aware splitting
• Metadata
• Top-k retrieval
• Hybrid search
• Reranking

Microsoft specifically notes that teams can compare search algorithms, top-k values and chunk sizes using retrieval evaluation to determine which configuration gives better results.

ZA Technologies’ RAG / LLM Integration service is a natural internal link here because RAG performance depends on the full knowledge architecture, including data preparation, retrieval and model integration.

Test Multi-Step Questions

Simple FAQ questions are the easiest RAG scenario.

Production users often ask questions requiring information from multiple places.

For example:

“We are a Canadian company with 120 employees. Which plan includes SSO, audit logs and Canadian data residency?”

The system may need to retrieve:

• Pricing documentation
• Authentication features
• Security documentation
• Hosting region information

Your evaluation should measure whether the system retrieves all required evidence and combines it correctly.

Do not mark the test as successful simply because one part of the answer is correct.

Verify Citations Separately

Displaying citations makes a chatbot look trustworthy.

That does not mean the citations are trustworthy.

Test whether:

1. The cited document actually supports the answer.
2. The correct section was retrieved.
3. The citation is attached to the correct claim.
4. Old or irrelevant documents are not being presented as evidence.

A chatbot that gives an unsupported answer and attaches a legitimate-looking document can create more trust than it deserves.

Citation correctness deserves its own evaluation.

Create a RAG Launch Scorecard

Before launch, your team can use a simple scorecard:

Set pass criteria before testing.

For example:

Critical policy questions: 100% correct source retrieval

Groundedness: 95%+

Unsupported-answer rate: below agreed threshold

Cross-permission retrieval: zero tolerance

The exact targets will depend on the risk of your application.

A marketing assistant and a financial-policy chatbot should not necessarily have the same launch threshold.

Human Review Still Matters

Automated evaluators are useful because they can test hundreds or thousands of examples quickly.

They should not completely replace human evaluation.

Have subject-matter experts review samples involving:

• High-risk policies
• Financial information
• Compliance
• Complex reasoning
• Ambiguous questions
• Poor automated scores
• Customer complaints

A good evaluation program combines automated measurement with human judgment.

The goal is not to produce one impressive RAG accuracy number.

It is to understand where the system is reliable and where it is not.

For broader launch testing, internally link to ZA Technologies’ Quality Engineering & Testing services and its QA Testing Before Launch guide. AI applications still need traditional software QA, but they also require answer-quality evaluation that normal functional testing does not cover.

Do Not Stop Evaluating After Launch

Your RAG system changes even when the model does not.

Someone updates a policy.

New documents are added.

A page changes structure.

An embedding model changes.

Your customers begin asking questions you never predicted.

Evaluation should therefore continue after launch.

Monitor:

• Low-confidence queries
• Failed searches
• User corrections
• Escalations
• Frequently unanswered questions
• Poorly rated answers
• New query patterns

ZA Technologies’ AI Chatbots & Agents service includes ongoing monitoring and refinement because conversational systems improve when production behavior feeds back into testing.

Turn real failures into new evaluation cases.

If a customer exposes a weakness once, your automated test suite should make sure the same weakness does not quietly return later.

Final Thoughts

A RAG system is not ready for production because it gave ten impressive demo answers.

It is ready when your team understands how it behaves when retrieval is strong, weak, incomplete, conflicting or completely missing.

Test the retriever.

Test the answer.

Test groundedness.

Test completeness.

Test citations.

Test questions that cannot be answered.

Test messy customer language.

Test outdated documents.

Test multiple-document questions.

Then repeat the evaluation every time your knowledge base, retrieval configuration or model changes.

The most important question before launch is not:

“Does our RAG chatbot work?”

Ask:

“Do we know exactly when it works, why it works and how it behaves when the evidence is not good enough?”

That is the difference between a convincing AI demo and a RAG system customers can actually trust.