Clean text wins
Your retrieval system searches chunks, not whole PDFs. Good chunks come from clean text with stable source names, page numbers, and section titles.
If the text is messy, embeddings and prompts will also be messy. Prepare the documents before you build the index.
What to prepare
Use 3 to 6 sample documents for your first version. Pick documents with real value: college placement rules, club handbook, hostel FAQ, startup policy, product docs, or a small research note.
Keep originals in one folder. Store extracted text in another folder. This makes your repo easy to inspect later.
Folder layout
Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.
0 of 5 done
Chunk format
Save chunks as JSONL. Each line is one searchable chunk with text and metadata.
Metadata is what lets you show citations later. Keep it boring and consistent.
{"chunk_id":"placement_rules_2024__p03__c01","source":"placement_rules_2024.pdf","page":3,"section":"Eligibility","text":"Students with one active backlog are not eligible for Day 1 companies unless the company policy allows it."}
{"chunk_id":"placement_rules_2024__p04__c02","source":"placement_rules_2024.pdf","page":4,"section":"Offer rules","text":"A student who accepts an offer through campus placement cannot sit for lower CTC companies later."}Extract the text
Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.
0 of 5 done
Small extraction script
This script extracts text from text-based PDFs using pypdf. It also handles .txt and .md files.
After running it, open the output files and inspect them. Do not trust extraction blindly.
# pip install pypdf
from pathlib import Path
from pypdf import PdfReader
RAW = Path("data/raw")
OUT = Path("data/processed_text")
OUT.mkdir(parents=True, exist_ok=True)
for path in RAW.iterdir():
if path.suffix.lower() == ".pdf":
reader = PdfReader(str(path))
parts = []
for i, page in enumerate(reader.pages, start=1):
text = page.extract_text() or ""
parts.append(f"\n\n[PAGE {i}]\n{text}")
(OUT / f"{path.stem}.txt").write_text("".join(parts), encoding="utf-8")
elif path.suffix.lower() in [".txt", ".md"]:
text = path.read_text(encoding="utf-8")
(OUT / f"{path.stem}.txt").write_text(text, encoding="utf-8")
print("done")Chunk by meaning
A chunk should usually hold one idea, not one random slice. For many QA apps, 300 to 800 words per chunk is a sensible starting point.
Try to split on headings, FAQ questions, policy points, or paragraphs. Keep overlap small when a paragraph continues across a boundary.
Avoid giant chunks
A full document as one chunk looks easy, but it hurts search. The retriever may return too much unrelated text, and the answer model may miss the exact line.
Tiny chunks are also weak because they lose context. A line like “This rule applies from July” is useless without the rule before it.
Do this now
Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.
0 of 5 done