Model Evaluation - lesson 4 | HackIndia Skills

Key point

One clean rule

Your validation set must act like future data. Do not let the same example, same user, or same source item appear in both training and validation.

If your split is messy, your score may look good in the demo and fail when someone tries new data.

What validation means

Training data teaches the model. Validation data checks the model while you are building it.

For a hackathon project, a common split is 80 percent train and 20 percent validation. The exact ratio is less important than making the split before tuning the model.

Your to-do

Make the split

Do each one yourself, then tap it to tick it off. The ticks are only a checklist for you: they are not marked or scored.

Put your dataset in one table with features and the target label.
Split once into train and validation before model tuning.
Use stratify for classification so each class keeps a similar ratio.
Set a random_state so you can reproduce the same split.
Report the split method in your project description.

0 of 5 done

Example

Simple sklearn split

This works for a normal classification dataset where each row is independent. For example, predicting whether a support ticket is urgent or not urgent.

Check the class counts after splitting. If your validation set has almost no positive cases, your precision or recall will not mean much.

from sklearn.model_selection import train_test_split

X = df.drop(columns=["label"])
y = df["label"]

X_train, X_val, y_train, y_val = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

print("Train size:", len(X_train))
print("Validation size:", len(X_val))
print("Train class ratio:")
print(y_train.value_counts(normalize=True))
print("Validation class ratio:")
print(y_val.value_counts(normalize=True))

Common mistake

Common leakage

Leakage means the validation set contains information the model would not have in real use. It makes results look better than they are.

A common college project mistake is splitting image files after augmentation. If the original image is in train and a rotated copy is in validation, the model is being tested on something it has almost seen.

Tip

Split by group

If many rows come from the same person, shop, hospital, device, or YouTube video, split by that group. This gives a more honest test.

Example: for a student attendance model, do not put the same student in both train and validation. The model may learn that student, not attendance patterns.