Pandas - lesson 4 | HackIndia Skills

Key point

First inspect, then clean

Do not start by replacing values. First learn the shape, column types, missing values, duplicates, and strange ranges.

Your repo should show this thinking. A simple quality report proves you did not clean blindly.

What to inspect

Shape tells you how many rows and columns you loaded. Columns tell you what fields exist. Dtypes tell you how pandas understands each column.

In pandas 3.0, text columns are inferred as the dedicated str dtype by default. Older tutorials may call them object columns, so do not copy that assumption blindly.

Your to-do

Basic checks

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Use shape to confirm row and column count.
Use columns to check names, spelling, and unexpected spaces.
Use head and tail to view real rows from both ends.
Use info to see dtypes and non-missing counts.
Use describe to inspect ranges for numeric columns.

0 of 5 done

Example

Inspection script

Put this in a notebook or a small Python file after loading your raw data. It works well for a college placement dataset, UPI transactions, product orders, or any tabular CSV.

import pandas as pd

students = pd.read_csv("data/raw/students.csv")

print("Shape:", students.shape)
print("Columns:", students.columns.tolist())

print("\nFirst rows")
print(students.head())

print("\nLast rows")
print(students.tail())

print("\nInfo")
students.info()

print("\nNumeric summary")
print(students.describe())

print("\nMissing values")
print(students.isna().sum().sort_values(ascending=False))

print("\nDuplicate rows:", students.duplicated().sum())

Tip

Use isna

Use isna() or notna() to detect missing values. pandas treats None as missing too.

Do not check missing values with equality. Missing sentinels do not behave like normal values, so df[col] == None or value == np.nan can mislead you.

Common mistake

Trusting dtypes too much

A column can have a dtype and still be wrong for your project. For example, a fees column may be str because values include ₹, commas, or words like pending.

Dates also often load as text first. In this lesson, just find the issue and record it. You will fix it in the cleaning lesson.