All the reading

Pandas

8 lessons, about 33 minutes. The check's questions come from these.

Setting up and loading files

1. Why Cleaning Is Most Work

4 min

Key point

Main idea

Your job is not to make pandas run. Your job is to make messy data safe enough to trust.

Most project bugs come from wrong types, missing values, duplicate rows, broken dates, mixed spellings, and tables that do not join cleanly.

Why it takes time

Real data is collected by forms, apps, admins, vendors, sensors, and people in a hurry. A college event sheet may have “CSE”, “cse”, “Computer Science”, and blank department values in the same column.

pandas is used because it works well with tables. A table in pandas is called a DataFrame. You use it to load, explore, clean, process, join, group, and write data.

Example

Messy looks normal

This tiny CSV already has problems. It looks usable, but it is not ready for analysis.

student_id,name,dept,fee_paid,joined_on
101,Asha,CSE,500,2026-08-01
102,Rahul,cse,,01/08/26
102,Rahul,CSE,500,2026-08-01
103,Meena,Computer Science,₹500,
104,,ECE,no,2026/08/03

Your to-do

What you check

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Check column names and what each row means.
Check missing values in important columns.
Check data types like dates, numbers, and text.
Check duplicate rows and duplicate IDs.
Check category spelling, casing, and units.
Check whether tables can join using a stable key.

0 of 6 done

2. Set Up Pandas and NumPy

4 min

Key point

One clean setup

Use one project folder and one Python environment for this track. It keeps your cleaning code, raw files, outputs, and dependency notes together.

pandas gives you DataFrames for table work. NumPy supports pandas and gives you arrays, missing-value sentinels, and fast vector operations.

What you need

For this track, your base stack is pandas 3.0.6 and NumPy 2.5.3. pandas is the main tool for loading, inspecting, cleaning, grouping, joining, and saving tabular data.

A DataFrame is pandas’ table object. Think of it like a spreadsheet or database table you can control with code.

Your to-do

Make the folder

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Create a folder named clean-data-with-pandas.
Create subfolders named data_raw, data_clean, notebooks, and charts.
Create a Python file named check_setup.py.
Keep this folder as the start of your public HackIndia repo.

0 of 4 done

3. Load CSV and JSON Files

4 min

Key point

One loading idea

pandas loads tabular data into a DataFrame. A DataFrame is the table you will inspect, clean, join, group, chart, and export later.

Use read_* functions to load files, such as read_csv() and read_json(). Use to_* methods later when you write cleaned output.

Pick raw files

For your final HackIndia repo, keep raw files unchanged. Put them in a data/raw folder so anyone can rerun your notebook or script.

Good practice is to load at least two files. Example: orders.csv and customers.json for a small UPI payments, canteen, fest, or placement dataset.

Example

Load a CSV

read_csv() reads a CSV path and returns a DataFrame. pandas detects common missing-value markers by default, so blanks often become missing values automatically.

Do not clean inside the raw file. Load first, then clean in code in later lessons.

Inspecting and cleaning columns

4. Inspect Shape Types and Quality

4 min

Key point

First inspect, then clean

Do not start by replacing values. First learn the shape, column types, missing values, duplicates, and strange ranges.

Your repo should show this thinking. A simple quality report proves you did not clean blindly.

What to inspect

Shape tells you how many rows and columns you loaded. Columns tell you what fields exist. Dtypes tell you how pandas understands each column.

In pandas 3.0, text columns are inferred as the dedicated str dtype by default. Older tutorials may call them object columns, so do not copy that assumption blindly.

Your to-do

Basic checks

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Joining tables and shipping work

7. Join Tables and Check Charts

4 min

Key point

One idea

A join is not done when the code runs. It is done when the row counts, missing matches, and chart shape make sense.

Why joins fail

Most data projects have more than one table. A college events dataset may have registrations in one CSV and college details in another JSON file.

The risky part is the join key. If one file has college_id as 101 and another has "101 ", pandas may not match them until you clean type and spaces.

Your to-do

Pick the join

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Use an inner join when you only want rows found in both tables.
Use a left join when your main table must keep all its rows.
Use clear key columns, like student_id, order_id, college_id, or pincode.
Check duplicates in the key before joining.
Compare row counts before and after the merge.

0 of 5 done

10 questions, 7 to pass.

Tip

Use AI carefully

You can ask ChatGPT, Gemini, or Claude to review a sample of raw data and suggest cleaning checks. This saves time when column names are unclear.

But verify every suggestion. AI may assume meanings, invent missing business rules, or miss local details like rupee symbols, Indian date formats, or mixed phone number formats.

Prompt to use:
I am cleaning this dataset for a pandas project. List likely data quality issues and checks I should run. Do not write final code yet. Here are 10 sample rows:
[paste rows]

Example

Install and check

Install pandas and NumPy inside your project environment. If your college laptop already has Python, use the option your setup supports.

pip is the common option. conda is also fine if you already use it.

pip install pandas
pip install numpy

# conda option
conda install -c conda-forge pandas
pip install numpy

Example

Verify imports

Put this in check_setup.py and run it. This proves your project can import the two libraries before you start cleaning real files.

Do not skip this. It catches broken installs early, before you are rushing near a hackathon deadline.

import pandas as pd
import numpy as np

print("pandas", pd.__version__)
print("numpy", np.__version__)

sample = pd.DataFrame({
    "city": ["Pune", "Bengaluru", "Delhi"],
    "orders": [120, 180, 95],
    "revenue": np.array([24000, 54000, 19000])
})

print(sample)
print(sample.dtypes)

Example

Load normal JSON

JSON can be shaped in different ways. A common beginner-friendly shape is a list of records, where each record becomes one row.

If AI generates sample JSON for you, check that it is valid JSON, uses double quotes, and has the same keys in most records.

[
  {"customer_id": 1, "name": "Asha", "city": "Pune"},
  {"customer_id": 2, "name": "Rahul", "city": "Jaipur"}
]

import pandas as pd

customers = pd.read_json("data/raw/customers.json")

print(customers.head())
print(customers.shape)
print(customers.columns)

Example

Load JSON lines

Some APIs and data exports store one JSON object per line. This is called line-delimited JSON or JSON Lines.

For this format, pass lines=True. If you also use chunksize, pandas returns an iterator instead of loading the whole file at once.

{"order_id": 101, "amount": 249, "status": "paid"}
{"order_id": 102, "amount": 99, "status": "failed"}

import pandas as pd

orders_log = pd.read_json("data/raw/orders_log.jsonl", lines=True)

for chunk in pd.read_json("data/raw/orders_log.jsonl", lines=True, chunksize=1000):
    print(chunk.shape)

Tip

Keep loading boring

Create one small loading script before you start cleaning. It should load each file and print head, shape, and columns.

This gives reviewers confidence that your project runs. It also helps you catch wrong paths before you waste time debugging cleaning code.

import pandas as pd

orders = pd.read_csv("data/raw/orders.csv")
customers = pd.read_json("data/raw/customers.json")

print("orders", orders.shape, list(orders.columns))
print(orders.head())

print("customers", customers.shape, list(customers.columns))
print(customers.head())

Example

Inspection script

Put this in a notebook or a small Python file after loading your raw data. It works well for a college placement dataset, UPI transactions, product orders, or any tabular CSV.

import pandas as pd

students = pd.read_csv("data/raw/students.csv")

print("Shape:", students.shape)
print("Columns:", students.columns.tolist())

print("\nFirst rows")
print(students.head())

print("\nLast rows")
print(students.tail())

print("\nInfo")
students.info()

print("\nNumeric summary")
print(students.describe())

print("\nMissing values")
print(students.isna().sum().sort_values(ascending=False))

print("\nDuplicate rows:", students.duplicated().sum())

Example

Quality report

Create a small report from each raw file. Commit it in your repo as reports/quality_notes.txt or print it from scripts/inspect.py.

This report is not a fancy dashboard. It is proof that you inspected the data before changing it.

# reports/quality_notes.txt
File: data/raw/students.csv
Shape: 500 rows, 8 columns
Possible key column: student_id
Missing values: email has 23, graduation_year has 4
Duplicate rows: 2
Type issues: fees is str, expected numeric rupees
Range issues: attendance_percent has values above 100
Notes: clean column names before joining with placements.csv

5. Clean Missing and Bad Values

5 min

Key point

One rule

Do not guess missing values by checking if a cell equals np.nan, pd.NA, or NaT. Use isna() and notna().

In pandas, missing values can look different by column type. None is also treated as missing.

What to clean

Messy files often have blanks, wrong spellings, impossible numbers, and mixed formats. For example, fees can be empty, negative, or written as "Rs 500" in one row and 500 in another.

Your job is not to make the data look perfect. Your job is to make clear, repeatable choices that another builder can run again.

Your to-do

Pick a fix

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Use isna() to count missing values before changing anything.
Use fillna() when a safe default exists, like 0 orders for a customer.
Use dropna() when a row cannot be used without that value, like missing user_id.
Use replace() or mask() for bad values like negative age or "N/A" text.
Convert types after cleaning text symbols, spaces, and invalid entries.

0 of 5 done

Common mistake

Avoid chained assignment

Older tutorials often show chained assignment. In pandas 3.0, Copy-on-Write means that pattern no longer works as people expect.

Use .loc[row_condition, column_name] = value when you update part of a DataFrame.

# Avoid this pattern
# df[df["fee"] < 0]["fee"] = pd.NA

# Use this instead
df.loc[df["fee"] < 0, "fee"] = pd.NA

Example

Clean a fees file

This example cleans common college fest registration data. It finds missing names, fixes bad fee values, standardises city text, and removes rows with no student_id.

Notice the order. Clean symbols first, convert to number, then handle missing and bad values.

import pandas as pd
import numpy as np

raw = pd.DataFrame({
    "student_id": [101, 102, None, 104, 105],
    "name": ["Asha", "", "Ravi", None, "Meena"],
    "city": [" Pune ", "MUMBAI", "mumbai", "Delhi", None],
    "fee": ["₹500", "", "-100", "Rs 300", None]
})

# Check missing values
print(raw.isna().sum())

clean = raw.copy()

# Treat empty names as missing
clean["name"] = clean["name"].replace("", pd.NA)

# Standardise city text
clean["city"] = clean["city"].str.strip().str.title()
clean["city"] = clean["city"].fillna("Unknown")

# Clean fee text and convert
clean["fee"] = (
    clean["fee"]
    .astype("str")
    .str.replace("₹", "", regex=False)
    .str.replace("Rs", "", regex=False)
    .str.strip()
)
clean["fee"] = pd.to_numeric(clean["fee"], errors="coerce")

# Mark impossible fees as missing
clean.loc[clean["fee"] < 0, "fee"] = pd.NA

# Fill safe default for missing fee
clean["fee"] = clean["fee"].fillna(0)

# Drop rows without key ID
clean = clean.dropna(subset=["student_id"])

print(clean)

Tip

Write your reason

For each cleaning choice, add a short comment. Your future internship reviewer should know why you filled, dropped, or replaced a value.

If you use AI to suggest cleaning code, ask it to list assumptions. Then check every assumption against your raw file.

Missing affects charts

pandas calculations like mean usually skip missing values by default. That is useful, but it can hide how much data was missing.

Before a chart, decide what should happen to missing values. Use fillna() or dropna() if the default chart behaviour is not what you want.

Your to-do

Do this now

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Open one raw file from your project idea or practice dataset.
Print missing counts with df.isna().sum().
Choose one column to fill, one to drop from, and one to standardise.
Update values with .loc when using a row condition.
Save your cleaning code in a script or notebook with comments.

0 of 5 done

6. Filter Group and Summarise

4 min

Key point

One idea

Cleaning is not only fixing bad values. You also reduce the table to the rows and summaries your project needs.

Use filters to choose rows. Use groupby to split rows into groups, calculate numbers, and combine the result into a new table.

When to filter

Filter when a row should not be part of an answer. For example, keep only paid orders, completed rides, valid attendance records, or students from one department.

A filter should be written as a clear condition. If you cannot explain the condition in one sentence, split it into smaller filters.

Example

Filter rows

This example keeps completed UPI orders of ₹100 or more. Notice the brackets around each condition. They matter when you combine conditions.

import pandas as pd

orders = pd.read_csv("data/raw/orders.csv")

paid_big_orders = orders[
    (orders["status"] == "paid") &
    (orders["payment_mode"] == "UPI") &
    (orders["amount"] >= 100)
]

print(paid_big_orders.shape)
print(paid_big_orders.head())

Your to-do

Build a filter

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Pick the business question, such as paid orders by city.
Find the columns needed for the condition.
Write one condition first and print the row count.
Add the next condition with & for AND or | for OR.
Save the filtered table into a new variable.

0 of 5 done

Group and summarise

Grouping is useful when raw rows are too detailed. Instead of reading 20,000 order rows, you can create one row per city, college, product, or date.

pandas calls this split-apply-combine. It splits rows by a column, applies calculations, then returns a smaller result.

Example

Group orders

Here you calculate order count, total revenue, average amount, and highest order value for each city. This kind of table is useful in your final repo because it proves you can turn raw data into project-ready data.

city_summary = (
    paid_big_orders
    .groupby("city")
    .agg(
        orders_count=("order_id", "count"),
        total_revenue=("amount", "sum"),
        avg_order_value=("amount", "mean"),
        max_order_value=("amount", "max")
    )
    .reset_index()
    .sort_values("total_revenue", ascending=False)
)

print(city_summary.head(10))

city_summary.to_csv("data/clean/city_summary.csv", index=False)

Common mistake

Avoid chained assignment

Older tutorials often change filtered data in one line after slicing. In pandas 3.0, Copy-on-Write means chained assignment no longer works the way old tutorials expect.

Use a new variable, then assign with .loc when you need to edit rows. This keeps your notebook or script easier to debug.

# Avoid patterns like this from old tutorials:
# orders[orders["status"] == "paid"]["amount"] = 0

paid_orders = orders[orders["status"] == "paid"].copy()
paid_orders.loc[:, "amount_after_fee"] = paid_orders["amount"] - 2

Tip

Use AI carefully

You can ask an AI tool to suggest groupby summaries for your dataset. Paste column names and your project question, not private data.

Check the returned code for three things: correct column names, correct aggregation meaning, and no chained assignment copied from older pandas examples.

Your to-do

Do this now

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Open the cleaned file from lesson 5.
Create one filtered table linked to your project question.
Create one grouped summary with at least three aggregations.
Print the top 10 rows and check if the numbers make sense.
Save the summary as a CSV inside data/clean.

0 of 5 done

Example

Join and check

This example joins event registrations with college details. It keeps every registration, then checks which colleges did not match.

import pandas as pd

regs = pd.read_csv("data/clean_registrations.csv")
colleges = pd.read_json("data/clean_colleges.json")

# Clean join keys again before merge
regs["college_id"] = regs["college_id"].astype("str").str.strip()
colleges["college_id"] = colleges["college_id"].astype("str").str.strip()

print("regs rows:", len(regs))
print("college rows:", len(colleges))

merged = regs.merge(
    colleges,
    on="college_id",
    how="left",
    indicator=True
)

print("merged rows:", len(merged))
print(merged["_merge"].value_counts())

unmatched = merged[merged["_merge"] == "left_only"]
print(unmatched[["registration_id", "college_id"]].head())

merged.to_csv("output/merged_registrations.csv", index=False)

Common mistake

Avoid silent duplicates

If the right table has the same college_id many times, one registration can turn into many rows. Your total count will grow and your chart may look impressive but wrong.

Before merging, check whether the key should be unique. If it should be unique, remove or fix duplicates first.

duplicate_colleges = colleges[colleges.duplicated("college_id", keep=False)]
print(duplicate_colleges.sort_values("college_id").head())

Example

Chart the result

Quick charts are checks, not final dashboard work. Use them to catch odd joins, missing matches, and strange category counts.

Pandas plotting uses Matplotlib. For missing data, bar plots fill missing values with 0, while scatter plots and histograms drop missing values. Use fillna or dropna first if that is not what you want.

import matplotlib.pyplot as plt

# Count registrations by city after the join
city_counts = (
    merged["city"]
    .fillna("Unknown")
    .value_counts()
    .head(10)
)

city_counts.plot(kind="bar", title="Top cities by registrations")
plt.xlabel("City")
plt.ylabel("Registrations")
plt.tight_layout()
plt.savefig("output/top_cities.png")
plt.close()

# Check fee vs attendance only where both are present
plot_data = merged.dropna(subset=["fee_rupees", "attendance_percent"])
plot_data.plot(
    kind="scatter",
    x="fee_rupees",
    y="attendance_percent",
    title="Fee vs attendance"
)
plt.tight_layout()
plt.savefig("output/fee_vs_attendance.png")
plt.close()

8. Ship Your Cleaning Repo

4 min

Key point

The proof matters

Your repo is the proof, not your notebook screenshot. A reviewer should be able to open it, see the raw files, run your cleaning script, and find the cleaned output and charts.

Keep the project small. Two raw files are enough if you clean them properly and show one join or group result.

What to include

Use a public repository. Put data in clear folders so a classmate or recruiter does not have to guess where things are.

Your repo should show the full path from messy data to usable data: load, inspect, clean, join or group, chart, save.

Example

Good repo shape

This structure is simple and easy to review on a phone. Names can change, but keep the same idea.

clean-data-project/
  README.md
  requirements.txt
  data/
    raw/
      orders.csv
      customers.json
    cleaned/
      cleaned_orders.csv
      customer_summary.csv
  charts/
    orders_by_city.png
    payment_status_bar.png
  src/
    clean_data.py

Your to-do

Write your README

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

State the problem in 2 to 3 lines, like cleaning orders and customer data for a small D2C store.
List the raw files and what each row means.
Explain the cleaning done: missing values, bad types, duplicates, invalid rows, and renamed columns.
Mention the join or group result you created.
Add exact run steps and where outputs appear.

0 of 5 done

Example

README run steps

Do not write vague lines like run the code. Give commands and expected outputs. Use the same file names that exist in your repo.

Project: Clean order and customer data

Raw files:
- data/raw/orders.csv: one row per order
- data/raw/customers.json: one row per customer

Cleaning done:
- Standardised column names
- Converted order_date to datetime
- Filled missing city as Unknown
- Removed orders with negative amount
- Joined orders with customer city
- Created city-level order summary

How to run:
1. Install the packages in requirements.txt
2. Run: python src/clean_data.py
3. Check data/cleaned/ for CSV outputs
4. Check charts/ for PNG charts

Common mistake

Avoid hidden work

Do not clean data manually in Excel and then upload only the final CSV. That does not prove your pandas skill.

Do not commit secrets, private college data, phone numbers, or raw UPI details. If your dataset has personal data, remove or mask it before uploading.

Tip

Use AI carefully

You can ask an AI tool to review your README or suggest folder names. You can also paste your script and ask it to find unclear steps.

Check everything it returns. Make sure commands are real, file paths match your repo, and it does not invent charts, columns, packages, or results.

Your to-do

Do this now

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Create the repo folders: data/raw, data/cleaned, charts, and src.
Move at least two raw files into data/raw.
Save your final cleaning code as src/clean_data.py.
Run the script once and confirm cleaned CSV files and charts are created.
Write README.md with the problem, files, cleaning steps, run steps, and output locations.

0 of 5 done

Pandas

1. Why Cleaning Is Most Work

Main idea

Why it takes time

Messy looks normal

What you check

2. Set Up Pandas and NumPy

One clean setup

What you need

Make the folder

3. Load CSV and JSON Files

One loading idea

Pick raw files

Load a CSV

4. Inspect Shape Types and Quality

First inspect, then clean

What to inspect

Basic checks

7. Join Tables and Check Charts

One idea

Why joins fail

Pick the join

Common beginner mistake

Use AI carefully

Your proof target

Do this now

Install and check

Verify imports

Old tutorial traps

Use AI carefully

Do this now

Load normal JSON

Load JSON lines

Common loading mistakes

Keep loading boring

Do this now

Inspection script

Use isna

Trusting dtypes too much

Quality report

Do this now

5. Clean Missing and Bad Values

One rule

What to clean

Pick a fix

Avoid chained assignment

Clean a fees file

Write your reason

Missing affects charts

Do this now

6. Filter Group and Summarise

One idea

When to filter

Filter rows

Build a filter

Group and summarise

Group orders

Avoid chained assignment

Use AI carefully

Do this now

Join and check

Avoid silent duplicates

Use AI carefully

Chart the result

Do this now

8. Ship Your Cleaning Repo

The proof matters

What to include

Good repo shape

Write your README

README run steps

Avoid hidden work

Use AI carefully

Do this now