One idea
A RAG answer is good only if the cited source text supports it. Measure that directly with a few fixed questions, expected evidence, and a simple pass or fail.
What to measure
Do not start with fancy scores. For an internship-ready repo, show that you tested groundedness on your own sample documents.
Track four things: the question, the answer, the citations returned by your app, and whether the cited text actually supports the answer. If the answer adds facts not present in the sources, mark it fail.
Create a test set
Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.
0 of 5 done
Eval file example
Keep the format boring. It should be easy for a recruiter or teammate to read.
[
{
"id": "q1",
"question": "What is the refund deadline for the workshop?",
"expected_source": "sample_docs/workshop_policy.md",
"expected_evidence": "Refund requests must be submitted at least 7 days before the workshop date."
},
{
"id": "q2",
"question": "Does the policy mention hostel accommodation?",
"expected_source": null,
"expected_evidence": "Not answerable from the provided documents."
}
]Run the check
Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.
0 of 5 done
Tiny evaluator script
This script does not judge truth by itself. It creates a file where you can add pass or fail after reading the cited evidence.
import json
from pathlib import Path
# Replace this import with your app function.
# It should return: {"answer": "...", "citations": [{"source": "...", "text": "..."}]}
from app import answer_question
questions = json.loads(Path("eval_questions.json").read_text())
results = []
for item in questions:
output = answer_question(item["question"])
results.append({
"id": item["id"],
"question": item["question"],
"answer": output["answer"],
"citations": output["citations"],
"expected_source": item["expected_source"],
"expected_evidence": item["expected_evidence"],
"grounded": "TODO: pass/fail",
"notes": "TODO: explain using cited text"
})
Path("grounding_eval.json").write_text(json.dumps(results, indent=2))
print("Wrote grounding_eval.json")Common mistake
Do not mark an answer correct just because it sounds right. If the citation does not contain the claim, it is not grounded.
Also test one unanswerable question. A good document QA app should say it does not know from the provided documents instead of inventing an answer.
Use AI carefully
You can ask ChatGPT or Gemini to draft eval questions from your sample docs. Then check every question yourself against the document text.
Ask it: “Create 8 QA eval questions from this document. Include source lines and one unanswerable question.” Do not accept questions where the evidence is vague or missing.
Do this now
Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.
0 of 5 done