All the reading

RAG

7 lessons, about 32 minutes. The check's questions come from these.

Choosing and cleaning documents

1. Why retrieval beats memory

4 min

The problem

A chat model can sound confident even when it has not seen your document. If you ask about your college rulebook, internship policy, or a startup pitch deck, the model may answer from general patterns instead of your file.

For a document QA app, that is not enough. Your user needs answers that come from the uploaded documents and can be checked.

Key point

One idea

Do not ask the model to remember your documents. Retrieve the relevant parts first, then ask the model to answer only from those parts.

Memory vs retrieval

Model memory means the information learned during training, plus whatever is in the current chat. It is broad, but not reliable for your private PDFs, notices, invoices, resumes, or project docs.

Retrieval means your app searches your own document collection at answer time. It finds small pieces of text that match the question, puts them into the prompt, and asks the model to answer using that evidence.

Example

Tiny prompt difference

A memory-only prompt asks for an answer directly. A retrieval prompt includes evidence and clear rules.

Memory-only:
Question: What is the late submission penalty?
Answer the question.

Retrieval-grounded:
Evidence:
[doc: assignment_policy.txt, lines 12-14]
Late submissions lose 10% marks per day. Submissions more than 5 days late are not accepted.

Question: What is the late submission penalty?
Answer using only the evidence. If the evidence is not enough, say you do not know.

Common mistake

2. Prepare documents for search

5 min

Key point

Clean text wins

Your retrieval system searches chunks, not whole PDFs. Good chunks come from clean text with stable source names, page numbers, and section titles.

If the text is messy, embeddings and prompts will also be messy. Prepare the documents before you build the index.

What to prepare

Use 3 to 6 sample documents for your first version. Pick documents with real value: college placement rules, club handbook, hostel FAQ, startup policy, product docs, or a small research note.

Keep originals in one folder. Store extracted text in another folder. This makes your repo easy to inspect later.

Your to-do

Folder layout

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Chunking, embedding, and prompting

3. Create embeddings and a vector index

4 min

Key point

One idea

A vector index is a search table for meaning. You embed each document chunk once, store it with its source details, then embed the user question and fetch the closest chunks.

What you are building

In lesson 2, you prepared clean chunks. Now you will create a local vector index for those chunks.

Use Chroma for the local vector store. It is free, runs on your laptop, and saves the index to a folder you can rebuild from your repo.

Your to-do

Install packages

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Create a new Python virtual environment if you do not have one.
Install Chroma and an embedding library.
Keep your package list in requirements.txt.
Do not commit large generated index folders unless your project needs them.

0 of 4 done

Citations, evaluation, and shipping

5. Cite sources users can verify

5 min

Key point

One rule

A citation is useful only if a user can find the same evidence again. Cite the retrieved chunk, not the model’s memory or a vague file name.

For this track, a good citation has document name, page or section if available, and a chunk id or quoted line.

What to cite

Your answer should cite every important factual claim that comes from the documents. If the answer has three facts from two chunks, show both sources.

Do not cite general filler like “based on the documents”. Cite claims such as dates, fees, eligibility, deadlines, policy rules, product limits, and names.

Your to-do

Keep source metadata

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

10 questions, 7 to pass.

Example

Chunk format

Save chunks as JSONL. Each line is one searchable chunk with text and metadata.

Metadata is what lets you show citations later. Keep it boring and consistent.

{"chunk_id":"placement_rules_2024__p03__c01","source":"placement_rules_2024.pdf","page":3,"section":"Eligibility","text":"Students with one active backlog are not eligible for Day 1 companies unless the company policy allows it."}
{"chunk_id":"placement_rules_2024__p04__c02","source":"placement_rules_2024.pdf","page":4,"section":"Offer rules","text":"A student who accepts an offer through campus placement cannot sit for lower CTC companies later."}

Example

Small extraction script

This script extracts text from text-based PDFs using pypdf. It also handles .txt and .md files.

After running it, open the output files and inspect them. Do not trust extraction blindly.

# pip install pypdf
from pathlib import Path
from pypdf import PdfReader

RAW = Path("data/raw")
OUT = Path("data/processed_text")
OUT.mkdir(parents=True, exist_ok=True)

for path in RAW.iterdir():
    if path.suffix.lower() == ".pdf":
        reader = PdfReader(str(path))
        parts = []
        for i, page in enumerate(reader.pages, start=1):
            text = page.extract_text() or ""
            parts.append(f"\n\n[PAGE {i}]\n{text}")
        (OUT / f"{path.stem}.txt").write_text("".join(parts), encoding="utf-8")

    elif path.suffix.lower() in [".txt", ".md"]:
        text = path.read_text(encoding="utf-8")
        (OUT / f"{path.stem}.txt").write_text(text, encoding="utf-8")

print("done")

Example

Index your chunks

This example assumes you have a JSONL file from lesson 2. Each line has chunk text and metadata.

Use a small Sentence Transformers model so the project works without paid APIs. You can swap it later.

# index_chunks.py
import json
import chromadb
from chromadb.utils import embedding_functions

CHUNKS_FILE = "data/chunks.jsonl"
DB_DIR = "chroma_db"
COLLECTION = "doc_chunks"

embedding_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="all-MiniLM-L6-v2"
)

client = chromadb.PersistentClient(path=DB_DIR)
collection = client.get_or_create_collection(
    name=COLLECTION,
    embedding_function=embedding_fn
)

ids = []
documents = []
metadatas = []

with open(CHUNKS_FILE, "r", encoding="utf-8") as f:
    for i, line in enumerate(f):
        item = json.loads(line)
        ids.append(item.get("id", f"chunk-{i}"))
        documents.append(item["text"])
        metadatas.append({
            "source": item.get("source", "unknown"),
            "page": item.get("page", ""),
            "chunk_id": item.get("id", f"chunk-{i}")
        })

collection.add(ids=ids, documents=documents, metadatas=metadatas)
print(f"Indexed {len(ids)} chunks into {DB_DIR}")

Example

Search the index

Retrieval is the test. If your index cannot fetch useful chunks, the answer step will fail later.

Start with top 3 results. Read them. Check if they actually help answer the question.

# search.py
import chromadb
from chromadb.utils import embedding_functions

DB_DIR = "chroma_db"
COLLECTION = "doc_chunks"

embedding_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="all-MiniLM-L6-v2"
)

client = chromadb.PersistentClient(path=DB_DIR)
collection = client.get_collection(
    name=COLLECTION,
    embedding_function=embedding_fn
)

question = "What is the refund policy for late cancellation?"

results = collection.query(
    query_texts=[question],
    n_results=3
)

for rank, (doc, meta, dist) in enumerate(zip(
    results["documents"][0],
    results["metadatas"][0],
    results["distances"][0]
), start=1):
    print(f"\nResult {rank} | distance={dist:.4f}")
    print(f"Source: {meta['source']} page {meta.get('page', '')}")
    print(doc[:700])

4. Put evidence into the prompt

5 min

Key point

The core rule

A RAG answer is only as good as the context you put into the prompt. Treat retrieved chunks as evidence, not background decoration.

Your prompt must clearly separate instructions, evidence, and the user question. Then ask the model to answer only from the evidence.

What the prompt needs

After lesson 3, you have search results from a vector index. Each result should include chunk text and metadata such as file name, page, section, or chunk id.

In this lesson, your job is to pack the top chunks into a clean prompt. Do not paste the whole document. Use the few most relevant chunks that fit your model limit.

Your to-do

Build the prompt

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Collect the top 3 to 6 chunks returned by your retriever.
Label each chunk with a stable id like S1, S2, S3.
Include useful metadata such as file name and page number.
Tell the model to use only the provided evidence.
Tell the model what to do when the answer is not in the evidence.

0 of 5 done

Example

Prompt template

Use a template like this before calling your LLM. The important parts are the evidence labels and the fallback rule.

This example is plain Python. Connect it to whatever model call your app uses.

def build_prompt(question, retrieved_chunks):
    evidence_blocks = []

    for i, chunk in enumerate(retrieved_chunks, start=1):
        source_id = f"S{i}"
        file_name = chunk["metadata"].get("file", "unknown file")
        page = chunk["metadata"].get("page", "unknown page")
        text = chunk["text"].strip()

        evidence_blocks.append(
            f"[{source_id}] File: {file_name}, Page: {page}\n{text}"
        )

    evidence = "\n\n".join(evidence_blocks)

    return f"""
You are a document question-answering assistant.
Answer the user's question using only the evidence below.
If the evidence does not contain the answer, say: "I could not find this in the provided documents."
Do not use outside knowledge.
Keep the answer short and specific.

Evidence:
{evidence}

Question:
{question}

Answer:
""".strip()

Tip

Keep evidence small

More chunks do not always mean better answers. Too much context can bury the useful line and increase cost.

Start with top 4 chunks. If answers miss details, improve chunking or retrieval before increasing the number of chunks.

Common mistake

Do not trust wording

A model may still sound confident when the evidence is weak. Strong wording is not proof of grounding.

Make the prompt force uncertainty. The fallback sentence is part of your app, not a nice extra.

Example

Weak versus better

A weak prompt asks the model to answer from documents but does not show boundaries. A better prompt makes the evidence visible and numbered.

This also prepares your app for lesson 5, where answers will cite sources users can verify.

Weak:
Answer this question using the uploaded documents: What is the refund window?

Better:
Use only the evidence below.
If the answer is missing, say you could not find it.

[S1] File: policy.pdf, Page: 2
Refund requests must be raised within 7 days of payment.

Question: What is the refund window?
Answer:

Your to-do

Do this now

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Open the code where your retriever returns top chunks.
Add a build_prompt(question, retrieved_chunks) function.
Label chunks as S1, S2, S3 with file and page metadata.
Add the fallback line for unsupported questions.
Print one full prompt and check that evidence and question are clearly separated.

0 of 5 done

Example

Chunk record example

This is the kind of record your vector index should return. The exact database can change. The idea stays the same: text plus metadata.

{
  "id": "fee_policy_p2_c1",
  "text": "Students can pay semester fees by UPI, net banking, or demand draft. Late payment after 15 August has a fine of Rs 500.",
  "metadata": {
    "source": "college_fee_policy.pdf",
    "page": 2,
    "section": "Payment modes"
  }
}

Example

Prompt for citations

Ask the model to use only the retrieved evidence and to cite source ids inline. Keep the citation format simple so you can test it later.

You are answering questions using only the evidence below.
If the evidence does not contain the answer, say: I don’t know from the provided documents.
For every factual claim, add a citation in this format: [source: id, page].

Evidence:
1. id: fee_policy_p2_c1, source: college_fee_policy.pdf, page: 2
Text: Students can pay semester fees by UPI, net banking, or demand draft. Late payment after 15 August has a fine of Rs 500.

Question: Can I pay fees using UPI? What happens if I pay late?

Answer:

Example

Answer users can verify

A useful answer is not just correct. It tells the user where to check it.

For your repo, save a few example questions and cited answers. Recruiters should be able to open the sample document and verify the citation.

Yes. You can pay semester fees using UPI, net banking, or demand draft [source: fee_policy_p2_c1, page 2]. If you pay after 15 August, the late fine is Rs 500 [source: fee_policy_p2_c1, page 2].

Tip

Add a verifier

Before showing the answer, run a small check in code. Extract citation ids from the answer and confirm they are in the retrieved result ids.

This does not prove the answer is fully grounded. Lesson 6 handles evaluation. But it catches many broken citations.

import re

retrieved_ids = {"fee_policy_p2_c1", "hostel_rules_p1_c3"}
answer = "You can pay by UPI [source: fee_policy_p2_c1, page 2]."

cited_ids = set(re.findall(r"source:\s*([a-zA-Z0-9_\-]+)", answer))
missing = cited_ids - retrieved_ids

if missing:
    raise ValueError(f"Answer cites sources not retrieved: {missing}")

6. Measure grounded answers

5 min

Key point

One idea

A RAG answer is good only if the cited source text supports it. Measure that directly with a few fixed questions, expected evidence, and a simple pass or fail.

What to measure

Do not start with fancy scores. For an internship-ready repo, show that you tested groundedness on your own sample documents.

Track four things: the question, the answer, the citations returned by your app, and whether the cited text actually supports the answer. If the answer adds facts not present in the sources, mark it fail.

Your to-do

Create a test set

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Pick 5 to 10 questions from your sample documents.
Include direct questions, like fees, dates, rules, or eligibility.
Include 1 question that is not answerable from the documents.
Write the expected source file or chunk for each answer.
Save it as eval_questions.json in your repo.

0 of 5 done

Example

Eval file example

Keep the format boring. It should be easy for a recruiter or teammate to read.

[
  {
    "id": "q1",
    "question": "What is the refund deadline for the workshop?",
    "expected_source": "sample_docs/workshop_policy.md",
    "expected_evidence": "Refund requests must be submitted at least 7 days before the workshop date."
  },
  {
    "id": "q2",
    "question": "Does the policy mention hostel accommodation?",
    "expected_source": null,
    "expected_evidence": "Not answerable from the provided documents."
  }
]

Your to-do

Run the check

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Run each eval question through your QA app.
Save the answer and citations returned by the app.
Open the cited source text and compare it with the answer.
Mark pass if the cited text supports every important claim.
Mark fail if the answer guesses, overstates, or cites the wrong source.

0 of 5 done

Example

Tiny evaluator script

This script does not judge truth by itself. It creates a file where you can add pass or fail after reading the cited evidence.

import json
from pathlib import Path

# Replace this import with your app function.
# It should return: {"answer": "...", "citations": [{"source": "...", "text": "..."}]}
from app import answer_question

questions = json.loads(Path("eval_questions.json").read_text())
results = []

for item in questions:
    output = answer_question(item["question"])
    results.append({
        "id": item["id"],
        "question": item["question"],
        "answer": output["answer"],
        "citations": output["citations"],
        "expected_source": item["expected_source"],
        "expected_evidence": item["expected_evidence"],
        "grounded": "TODO: pass/fail",
        "notes": "TODO: explain using cited text"
    })

Path("grounding_eval.json").write_text(json.dumps(results, indent=2))
print("Wrote grounding_eval.json")

Common mistake

Do not mark an answer correct just because it sounds right. If the citation does not contain the claim, it is not grounded.

Also test one unanswerable question. A good document QA app should say it does not know from the provided documents instead of inventing an answer.

Tip

Use AI carefully

You can ask ChatGPT or Gemini to draft eval questions from your sample docs. Then check every question yourself against the document text.

Ask it: “Create 8 QA eval questions from this document. Include source lines and one unanswerable question.” Do not accept questions where the evidence is vague or missing.

Your to-do

Do this now

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Create eval_questions.json with at least 5 questions.
Add 1 unanswerable question to test refusal.
Run your app and generate grounding_eval.json.
Manually mark each result pass or fail with a short note.
Commit both eval files to your public repo.

0 of 5 done

7. Ship the repo end to end

4 min

Key point

Repo proves the skill

Your app is not shipped when it works only on your laptop. It is shipped when a recruiter or hackathon judge can clone it, run it, inspect the code, and see proof that answers are grounded.

What to include

Your public repo should show the full path from documents to cited answers. Do not hide the important parts in screenshots or private notebooks.

Keep sample documents small and safe to publish. Use your own notes, public policy pages, college handbook excerpts, or dummy startup docs. Do not upload private PDFs, paid course material, or anyone’s Aadhaar, marksheet, or phone number.

Your to-do

Minimum repo checklist

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Add app code for loading documents, retrieval, prompting, answer generation, citations, and evaluation.
Add setup steps that start from a fresh clone, including Python version, install command, and environment variables.
Add sample documents in a clearly named folder.
Add example questions with expected cited answers.
Add a short grounding evaluation with results and notes on failures.
Add a licence if you want others to reuse the code.

0 of 6 done

Example

Simple repo shape

Use boring names. They help people inspect your project fast.

doc-qa-rag/
  README.md
  requirements.txt
  .env.example
  app.py
  ingest.py
  evaluate.py
  data/
    sample_docs/
      college_faq.txt
      refund_policy.txt
  examples/
    questions.json
    cited_answers.md
  eval/
    grounding_eval.md

Example

README skeleton

Your README is the front door. Write it for someone who has five minutes before their next meeting.

# Document QA RAG App

Ask questions from the sample documents and get answers with citations.

## Features
- Loads documents from data/sample_docs
- Splits documents and creates embeddings
- Retrieves evidence for each question
- Answers with source citations
- Includes a small grounding evaluation

## Setup
1. Clone the repo
2. Create a virtual environment
3. Install dependencies: pip install -r requirements.txt
4. Copy .env.example to .env and add your API key
5. Run ingestion: python ingest.py
6. Start the app: python app.py

## Try these questions
- What is the refund window?
- Who can apply for the hostel Wi-Fi pass?

## Grounding evaluation
See eval/grounding_eval.md for questions, answers, citations, and notes.

Common mistake

Do not overclaim

A small honest evaluation is better than a big vague claim. Say what works, what fails, and what you would fix next.

For example, write: “The app answers direct policy questions well. It struggles when the answer needs two far-apart sections.” This sounds more credible than “highly accurate RAG system”.

Tip

Use AI carefully

You can ask ChatGPT, Gemini, or Claude to review your README, write setup steps, or suggest evaluation questions. Paste your folder tree and explain how the app runs.

Check every AI suggestion. Make sure commands are real, file names match your repo, and the sample outputs actually came from your app.

Your to-do

Do this now

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Create the repo folders: data/sample_docs, examples, and eval.
Write a README with setup, run commands, sample questions, and evaluation link.
Add .env.example instead of committing real API keys.
Run the app from a fresh terminal and fix missing steps.
Push to a public GitHub repo and submit the link on HackIndia.

0 of 5 done

RAG

1. Why retrieval beats memory

The problem

One idea

Memory vs retrieval

Tiny prompt difference

2. Prepare documents for search

Clean text wins

What to prepare

Folder layout

3. Create embeddings and a vector index

One idea

What you are building

Install packages

5. Cite sources users can verify

One rule

What to cite

Keep source metadata

Common mistake

Think like an examiner

Basic RAG flow

Do this now

Chunk format

Extract the text

Small extraction script

Chunk by meaning

Avoid giant chunks

Do this now

Index your chunks

Search the index

Common mistake

Use AI carefully

Do this now

4. Put evidence into the prompt

The core rule

What the prompt needs

Build the prompt

Prompt template

Keep evidence small

Do not trust wording

Weak versus better

Do this now

Chunk record example

Prompt for citations

Answer users can verify

Avoid fake citations

Add a verifier

Do this now

6. Measure grounded answers

One idea

What to measure

Create a test set

Eval file example

Run the check

Tiny evaluator script

Common mistake

Use AI carefully

Do this now

7. Ship the repo end to end

Repo proves the skill

What to include

Minimum repo checklist

Simple repo shape

README skeleton

Do not overclaim

Use AI carefully

Do this now