Pandas - lesson 6 | HackIndia Skills

Key point

One idea

Cleaning is not only fixing bad values. You also reduce the table to the rows and summaries your project needs.

Use filters to choose rows. Use groupby to split rows into groups, calculate numbers, and combine the result into a new table.

When to filter

Filter when a row should not be part of an answer. For example, keep only paid orders, completed rides, valid attendance records, or students from one department.

A filter should be written as a clear condition. If you cannot explain the condition in one sentence, split it into smaller filters.

Example

Filter rows

This example keeps completed UPI orders of ₹100 or more. Notice the brackets around each condition. They matter when you combine conditions.

import pandas as pd

orders = pd.read_csv("data/raw/orders.csv")

paid_big_orders = orders[
    (orders["status"] == "paid") &
    (orders["payment_mode"] == "UPI") &
    (orders["amount"] >= 100)
]

print(paid_big_orders.shape)
print(paid_big_orders.head())

Your to-do

Build a filter

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Pick the business question, such as paid orders by city.
Find the columns needed for the condition.
Write one condition first and print the row count.
Add the next condition with & for AND or | for OR.
Save the filtered table into a new variable.

0 of 5 done

Group and summarise

Grouping is useful when raw rows are too detailed. Instead of reading 20,000 order rows, you can create one row per city, college, product, or date.

pandas calls this split-apply-combine. It splits rows by a column, applies calculations, then returns a smaller result.

Example

Group orders

Here you calculate order count, total revenue, average amount, and highest order value for each city. This kind of table is useful in your final repo because it proves you can turn raw data into project-ready data.

city_summary = (
    paid_big_orders
    .groupby("city")
    .agg(
        orders_count=("order_id", "count"),
        total_revenue=("amount", "sum"),
        avg_order_value=("amount", "mean"),
        max_order_value=("amount", "max")
    )
    .reset_index()
    .sort_values("total_revenue", ascending=False)
)

print(city_summary.head(10))

city_summary.to_csv("data/clean/city_summary.csv", index=False)

Common mistake

Avoid chained assignment

Older tutorials often change filtered data in one line after slicing. In pandas 3.0, Copy-on-Write means chained assignment no longer works the way old tutorials expect.

Use a new variable, then assign with .loc when you need to edit rows. This keeps your notebook or script easier to debug.

# Avoid patterns like this from old tutorials:
# orders[orders["status"] == "paid"]["amount"] = 0

paid_orders = orders[orders["status"] == "paid"].copy()
paid_orders.loc[:, "amount_after_fee"] = paid_orders["amount"] - 2

Tip

Use AI carefully

You can ask an AI tool to suggest groupby summaries for your dataset. Paste column names and your project question, not private data.

Check the returned code for three things: correct column names, correct aggregation meaning, and no chained assignment copied from older pandas examples.