Pandas - lesson 5 | HackIndia Skills

Key point

One rule

Do not guess missing values by checking if a cell equals np.nan, pd.NA, or NaT. Use isna() and notna().

In pandas, missing values can look different by column type. None is also treated as missing.

What to clean

Messy files often have blanks, wrong spellings, impossible numbers, and mixed formats. For example, fees can be empty, negative, or written as "Rs 500" in one row and 500 in another.

Your job is not to make the data look perfect. Your job is to make clear, repeatable choices that another builder can run again.

Your to-do

Pick a fix

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Use isna() to count missing values before changing anything.
Use fillna() when a safe default exists, like 0 orders for a customer.
Use dropna() when a row cannot be used without that value, like missing user_id.
Use replace() or mask() for bad values like negative age or "N/A" text.
Convert types after cleaning text symbols, spaces, and invalid entries.

0 of 5 done

Common mistake

Avoid chained assignment

Older tutorials often show chained assignment. In pandas 3.0, Copy-on-Write means that pattern no longer works as people expect.

Use .loc[row_condition, column_name] = value when you update part of a DataFrame.

# Avoid this pattern
# df[df["fee"] < 0]["fee"] = pd.NA

# Use this instead
df.loc[df["fee"] < 0, "fee"] = pd.NA

Example

Clean a fees file

This example cleans common college fest registration data. It finds missing names, fixes bad fee values, standardises city text, and removes rows with no student_id.

Notice the order. Clean symbols first, convert to number, then handle missing and bad values.

import pandas as pd
import numpy as np

raw = pd.DataFrame({
    "student_id": [101, 102, None, 104, 105],
    "name": ["Asha", "", "Ravi", None, "Meena"],
    "city": [" Pune ", "MUMBAI", "mumbai", "Delhi", None],
    "fee": ["₹500", "", "-100", "Rs 300", None]
})

# Check missing values
print(raw.isna().sum())

clean = raw.copy()

# Treat empty names as missing
clean["name"] = clean["name"].replace("", pd.NA)

# Standardise city text
clean["city"] = clean["city"].str.strip().str.title()
clean["city"] = clean["city"].fillna("Unknown")

# Clean fee text and convert
clean["fee"] = (
    clean["fee"]
    .astype("str")
    .str.replace("₹", "", regex=False)
    .str.replace("Rs", "", regex=False)
    .str.strip()
)
clean["fee"] = pd.to_numeric(clean["fee"], errors="coerce")

# Mark impossible fees as missing
clean.loc[clean["fee"] < 0, "fee"] = pd.NA

# Fill safe default for missing fee
clean["fee"] = clean["fee"].fillna(0)

# Drop rows without key ID
clean = clean.dropna(subset=["student_id"])

print(clean)

Tip

Write your reason

For each cleaning choice, add a short comment. Your future internship reviewer should know why you filled, dropped, or replaced a value.

If you use AI to suggest cleaning code, ask it to list assumptions. Then check every assumption against your raw file.

Missing affects charts

pandas calculations like mean usually skip missing values by default. That is useful, but it can hide how much data was missing.

Before a chart, decide what should happen to missing values. Use fillna() or dropna() if the default chart behaviour is not what you want.