Model Evaluation in Data Analysis

A good evaluation workflow separates model training from model testing and uses metrics appropriate to the problem.

• Measures how well predictions match real outcomes.
• Helps identify overfitting.
• Allows comparison between different models.
• Helps identify weaknesses in predictions.
• Supports better model selection.
• Shows whether the model is suitable for a business problem.

Suppose an online business wants to predict monthly sales from advertising spending.

import pandas as pd

data = {
    "Advertising": [10, 12, 15, 18, 20, 25, 28, 30, 35, 40],
    "Sales": [120, 135, 150, 175, 190, 220, 250, 270, 300, 340]
}

df = pd.DataFrame(data)

print(df)

A common approach is to train the model using one portion of the data and evaluate it on another portion.

from sklearn.model_selection import train_test_split

X = df[["Advertising"]]
y = df["Sales"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

MAE calculates the average absolute difference between actual values and predicted values.

from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(
    y_test,
    predictions
)

print("MAE:", mae)

If MAE is 12, the model's predictions differ from the actual values by 12 units on average for the evaluated observations.

MSE calculates the average squared prediction error. Squaring gives larger errors more influence.

from sklearn.metrics import mean_squared_error

mse = mean_squared_error(
    y_test,
    predictions
)

print("MSE:", mse)

RMSE is the square root of MSE. Because it is expressed in the same units as the target variable, it can be easier to interpret.

import numpy as np
from sklearn.metrics import mean_squared_error

rmse = np.sqrt(
    mean_squared_error(
        y_test,
        predictions
    )
)

print("RMSE:", rmse)

R², or the coefficient of determination, describes how much of the variation in the target is explained by the regression model relative to a baseline based on the mean target.

from sklearn.metrics import r2_score

r2 = r2_score(
    y_test,
    predictions
)

print("R² Score:", r2)

Comparing actual and predicted values helps analysts understand individual prediction errors.

comparison = pd.DataFrame({
    "Actual": y_test.values,
    "Predicted": predictions
})

comparison["Error"] = (
    comparison["Actual"]
    - comparison["Predicted"]
)

print(comparison)

A retail company predicts monthly sales from advertising expenditure. Model evaluation can help determine whether the prediction error is small enough for planning purposes.

• Marketing budget can be compared with predicted sales.
• Actual sales can be compared with predictions after each month.
• MAE can communicate average prediction error.
• RMSE can highlight larger prediction mistakes.
• R² can describe explanatory performance relative to a baseline.

Now suppose an online company wants to predict whether a customer will leave the service.

import pandas as pd

data = {
    "Age": [22, 25, 30, 35, 40, 45, 50, 55, 60, 65],
    "Monthly_Usage": [20, 25, 40, 45, 50, 30, 28, 22, 18, 15],
    "Churn": [1, 1, 0, 0, 0, 1, 1, 1, 1, 1]
}

df = pd.DataFrame(data)

print(df)

Accuracy is the proportion of predictions that are correct.

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(
    y_test,
    predictions
)

print("Accuracy:", accuracy)

A confusion matrix shows how classification predictions are distributed across actual and predicted classes.

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(
    y_test,
    predictions
)

print(cm)

Precision answers: Of the observations predicted as positive, how many were actually positive?

from sklearn.metrics import precision_score

precision = precision_score(
    y_test,
    predictions,
    zero_division=0
)

print("Precision:", precision)

Recall answers: Of all actual positive observations, how many did the model identify?

from sklearn.metrics import recall_score

recall = recall_score(
    y_test,
    predictions,
    zero_division=0
)

print("Recall:", recall)

F1 score is the harmonic mean of precision and recall.

from sklearn.metrics import f1_score

f1 = f1_score(
    y_test,
    predictions,
    zero_division=0
)

print("F1 Score:", f1)

The classification report provides precision, recall, F1 score, and support for each class.

from sklearn.metrics import classification_report

print(
    classification_report(
        y_test,
        predictions,
        zero_division=0
    )
)

A telecom company wants to identify customers who may leave the service.

The appropriate metric depends on the cost and consequences of different types of mistakes.

For binary classification, the ROC curve examines the trade-off between true positive rate and false positive rate across classification thresholds.

AUC summarizes the area under the ROC curve. It is commonly used as a threshold-independent ranking metric.

from sklearn.metrics import roc_auc_score

probabilities = model.predict_proba(
    X_test
)[:, 1]

auc = roc_auc_score(
    y_test,
    probabilities
)

print("ROC-AUC:", auc)

A precision-recall curve shows how precision and recall change as the classification threshold changes.

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(
    y_test,
    probabilities
)

print("Precision values:", precision)
print("Recall values:", recall)

Cross-validation repeatedly splits the available training data into training and validation portions to estimate model performance more robustly.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression

model = LinearRegression()

scores = cross_val_score(
    model,
    X,
    y,
    cv=5,
    scoring="r2"
)

print("Scores:", scores)
print("Mean R²:", scores.mean())

• Uses multiple train-validation splits.
• Provides several performance measurements.
• Helps assess variation across different data splits.
• Can support model comparison.
• Can be useful for selecting model configurations.

Suppose we want to compare Linear Regression and Random Forest Regression on the same evaluation data.

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

linear_model = LinearRegression()

forest_model = RandomForestRegressor(
    n_estimators=100,
    random_state=42
)

linear_model.fit(X_train, y_train)
forest_model.fit(X_train, y_train)

linear_pred = linear_model.predict(X_test)
forest_pred = forest_model.predict(X_test)

linear_mae = mean_absolute_error(
    y_test,
    linear_pred
)

forest_mae = mean_absolute_error(
    y_test,
    forest_pred
)

print("Linear Regression MAE:", linear_mae)
print("Random Forest MAE:", forest_mae)

The metric values can be examined alongside model complexity, interpretability, computational requirements, and business requirements.

A real-estate company predicts property prices using:

• Area
• Number of bedrooms
• Location-related features
• Property age
• Parking availability

MAE can communicate the average price error in currency units, while RMSE gives more influence to larger mistakes. R² can provide a complementary measure of explained variation.

A financial institution may use a classification model to predict whether an application belongs to an approval category based on application features.

Fraud detection datasets can be highly imbalanced, with legitimate transactions greatly outnumbering fraudulent ones.

In such cases, accuracy alone may hide poor detection of the minority class. Analysts often examine precision, recall, F1, ROC-AUC, and precision-recall behavior.

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score
)

print("Accuracy:",
      accuracy_score(y_test, predictions))

print("Precision:",
      precision_score(
          y_test,
          predictions,
          zero_division=0
      ))

print("Recall:",
      recall_score(
          y_test,
          predictions,
          zero_division=0
      ))

print("F1:",
      f1_score(
          y_test,
          predictions,
          zero_division=0
      ))

Some problems have more than two classes, such as predicting whether a support ticket is Billing, Technical, Account, or General.

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score
)

accuracy = accuracy_score(
    y_test,
    predictions
)

precision = precision_score(
    y_test,
    predictions,
    average="weighted",
    zero_division=0
)

recall = recall_score(
    y_test,
    predictions,
    average="weighted",
    zero_division=0
)

f1 = f1_score(
    y_test,
    predictions,
    average="weighted",
    zero_division=0
)

print("Accuracy:", accuracy)
print("Precision:", precision)
print("Recall:", recall)
print("F1:", f1)

Comparing training and testing performance can reveal potential generalization problems.

train_score = model.score(
    X_train,
    y_train
)

test_score = model.score(
    X_test,
    y_test
)

print("Training Score:", train_score)
print("Testing Score:", test_score)

Overfitting occurs when a model learns training-specific patterns that do not generalize well to unseen data.

Underfitting occurs when a model is too limited to capture important patterns in the data.

If both training and testing performance are poor, the analyst may investigate feature quality, model complexity, data quantity, and preprocessing.

A baseline is a simple reference model against which more complex models can be compared.

from sklearn.dummy import DummyRegressor

baseline = DummyRegressor(
    strategy="mean"
)

baseline.fit(
    X_train,
    y_train
)

baseline_pred = baseline.predict(
    X_test
)

print(
    "Baseline MAE:",
    mean_absolute_error(
        y_test,
        baseline_pred
    )
)

A sophisticated model should be evaluated against an appropriate baseline rather than judged only by its own metric value.

An HR analytics team predicts salary using years of experience, education-related variables, role, and other relevant features.

The team can evaluate the model using MAE and RMSE and compare results across validation folds. The evaluation should also consider whether the dataset and features appropriately represent the intended use.

A company automatically classifies support tickets into categories such as:

• Billing
• Technical Issue
• Account
• Shipping

The classification report can show precision, recall, and F1 for each category instead of relying only on overall accuracy.

Model comparison should use the same evaluation protocol wherever possible.

The following example demonstrates a complete classification evaluation workflow for customer churn prediction.

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    confusion_matrix,
    classification_report,
    roc_auc_score
)


# 1. Create dataset
df = pd.DataFrame({
    "Age": [22, 25, 28, 30, 35, 40, 45, 50, 55, 60],
    "Usage": [20, 25, 30, 45, 50, 40, 35, 25, 20, 15],
    "Churn": [1, 1, 1, 0, 0, 0, 1, 1, 1, 1]
})


# 2. Select features and target
X = df[["Age", "Usage"]]
y = df["Churn"]


# 3. Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)


# 4. Train the model
model = LogisticRegression()

model.fit(X_train, y_train)


# 5. Generate predictions
predictions = model.predict(X_test)

probabilities = model.predict_proba(
    X_test
)[:, 1]


# 6. Calculate metrics
accuracy = accuracy_score(
    y_test,
    predictions
)

precision = precision_score(
    y_test,
    predictions,
    zero_division=0
)

recall = recall_score(
    y_test,
    predictions,
    zero_division=0
)

f1 = f1_score(
    y_test,
    predictions,
    zero_division=0
)

auc = roc_auc_score(
    y_test,
    probabilities
)


# 7. Display results
print("Accuracy:", accuracy)
print("Precision:", precision)
print("Recall:", recall)
print("F1 Score:", f1)
print("ROC-AUC:", auc)

print("\nConfusion Matrix:")
print(
    confusion_matrix(
        y_test,
        predictions
    )
)

print("\nClassification Report:")
print(
    classification_report(
        y_test,
        predictions,
        zero_division=0
    )
)

Many classification models produce probabilities. The classification threshold can be changed depending on the application's objectives.

probabilities = model.predict_proba(
    X_test
)[:, 1]

threshold = 0.40

custom_predictions = (
    probabilities >= threshold
).astype(int)

print(custom_predictions)

Changing the threshold changes the balance between false positives and false negatives, so threshold selection should be evaluated against the actual business objective.

A model should not be evaluated from only one arbitrary train-test split when the dataset and problem require more robust analysis.

• Use cross-validation where appropriate.
• Inspect performance variation across folds.
• Check whether results change substantially with different splits.
• Evaluate on a final holdout test set after model selection.
• Monitor performance after deployment.

Evaluation does not necessarily end when a model is deployed. Real-world data can change over time.

Once actual outcomes become available, the organization can compare predictions with reality and monitor whether model performance changes.

Data leakage occurs when information that would not be available at prediction time improperly enters the training or evaluation process.

Preprocessing steps such as scaling or imputation should generally be fitted using training data within the evaluation workflow rather than using the complete dataset before splitting.

A Scikit-learn pipeline can combine preprocessing and model training so that preprocessing is performed consistently within cross-validation.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])

scores = cross_val_score(
    pipeline,
    X,
    y,
    cv=5,
    scoring="accuracy"
)

print("CV Scores:", scores)
print("Mean Accuracy:", scores.mean())

• Evaluating only on training data.
• Using accuracy for every classification problem.
• Ignoring class imbalance.
• Choosing a model using the test set repeatedly.
• Allowing data leakage during preprocessing.
• Comparing models using different evaluation datasets.
• Ignoring business costs of false positives and false negatives.
• Reporting one metric without explaining its context.
• Ignoring performance changes after deployment.

• Define the business objective before selecting metrics.
• Keep evaluation data separate from training data.
• Use a suitable baseline.
• Select metrics appropriate to the problem type.
• Use cross-validation when appropriate.
• Check for data leakage.
• Inspect multiple metrics rather than relying on one number.
• Examine individual prediction errors.
• Compare models using the same evaluation protocol.
• Monitor model performance after deployment.

Model Evaluation in Data Analysis

Python Data Analysis • Machine Learning • Real-Time Examples