AI & Machine Learning Case Study

A professional machine learning project is more than training a model. The complete lifecycle connects business requirements, data engineering, modeling, deployment, and continuous monitoring.

Business Questions

• Which customers are likely to leave?
• How early can churn risk be identified?
• Which customer behaviors indicate increased risk?
• How should the retention team prioritize customers?
• What is the financial impact of successful intervention?

Business Success Metrics

• Reduction in customer churn.
• Increase in customer retention.
• Revenue retained.
• Retention campaign response rate.
• Cost per successful intervention.

Assume the company provides historical customer data containing demographic, subscription, behavioral, and billing information.

Load the Data

import pandas as pd

df = pd.read_csv("customer_churn.csv")

print(df.head())
print(df.shape)
print(df.info())

Check Missing Values

missing = df.isnull().sum()

print(missing[missing > 0])

Check the Target Distribution

print(df["churned"].value_counts())
print(df["churned"].value_counts(normalize=True))

Why This Matters

• Missing data can affect model reliability.
• Duplicate records can distort training.
• Outliers can affect some algorithms.
• Class imbalance can make accuracy misleading.
• Data leakage can produce unrealistically strong results.

Select Features and Target

X = df.drop(columns=["customer_id", "churned"])
y = df["churned"]

Train/Test Split

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y
)

The training set is used to learn patterns, while the test set provides an estimate of how the model behaves on previously unseen customers.

Feature engineering transforms raw business information into signals that may help the model recognize patterns.

Example Features

• Average monthly usage.
• Support tickets per month.
• Payment failure rate.
• Recent login decline.
• Customer tenure group.
• Change in monthly spending.
• Number of products/services used.

df["tickets_per_month"] = (
    df["support_tickets"] /
    df["tenure_months"].clip(lower=1)
)

df["revenue_per_tenure"] = (
    df["monthly_charges"] *
    df["tenure_months"]
)

df["payment_risk"] = (
    df["payment_failures"] > 0
).astype(int)

A pipeline keeps preprocessing and model training together, reducing the risk of applying different transformations during training and prediction.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression

numeric_features = [
    "tenure_months",
    "monthly_charges",
    "support_tickets",
    "login_frequency",
    "payment_failures"
]

categorical_features = [
    "contract_type",
    "payment_method"
]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric_features),
    ("categorical", categorical_pipeline, categorical_features)
])

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000))
])

A professional workflow usually evaluates several candidate algorithms rather than assuming one algorithm will always be appropriate.

Example Random Forest

from sklearn.ensemble import RandomForestClassifier

rf_model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier(
        n_estimators=300,
        random_state=42,
        class_weight="balanced"
    ))
])

rf_model.fit(X_train, y_train)

rf_predictions = rf_model.predict(X_test)

Because churn prediction is a classification problem, several metrics should be considered together.

from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score
)

print(confusion_matrix(y_test, predictions))
print(classification_report(y_test, predictions))

auc = roc_auc_score(y_test, probabilities)
print("ROC-AUC:", auc)

• True Positive: Model correctly identifies a customer who churns.
• False Positive: Model flags a customer who does not churn.
• False Negative: Model misses a customer who churns.
• True Negative: Model correctly identifies a customer who remains.

Instead of simply producing a prediction, the application can convert probabilities into operational segments.

def risk_segment(probability):
    if probability >= 0.80:
        return "High Risk"
    elif probability >= 0.50:
        return "Medium Risk"
    else:
        return "Low Risk"

customer_predictions = pd.DataFrame({
    "customer_id": df.loc[X_test.index, "customer_id"],
    "churn_probability": probabilities
})

customer_predictions["risk_segment"] = (
    customer_predictions["churn_probability"]
    .apply(risk_segment)
)

In many business applications, stakeholders need more than a probability. They also need understandable reasons behind the prediction.

Potential Explanatory Signals

• Sharp decline in product usage.
• Multiple recent support complaints.
• Repeated payment failures.
• Short contract duration.
• High cost relative to usage.

Once a model has been validated, the complete pipeline can be serialized so that the same preprocessing and model are used during inference.

import joblib

joblib.dump(rf_model, "churn_model.joblib")

A web API can allow CRM applications or other business systems to request predictions.

from fastapi import FastAPI
import joblib
import pandas as pd

app = FastAPI()

model = joblib.load("churn_model.joblib")

@app.post("/predict")
def predict(customer: dict):

    data = pd.DataFrame([customer])

    probability = model.predict_proba(data)[0][1]

    return {
        "churn_probability": float(probability),
        "risk": (
            "high" if probability >= 0.80
            else "medium" if probability >= 0.50
            else "low"
        )
    }

Deploying a model is not the end of the project. Real-world data changes over time, and model performance can degrade.

Monitor

• Prediction volume.
• Prediction latency.
• Input data quality.
• Feature distributions.
• Prediction distribution.
• Actual business outcomes.
• Model performance over time.
• Data drift and concept drift.

New customer outcomes can become labeled data for future evaluation and retraining.

• AI: The system supports automated decision-making using learned patterns from customer behavior.
• Machine Learning: The prediction model learns relationships from historical labeled examples.
• Data Science: Data exploration, statistics, feature engineering, and evaluation are used to understand the problem.
• Software Engineering: The trained model is packaged into an application that can be deployed and maintained.
• MLOps: Monitoring, versioning, retraining, testing, and deployment keep the system reliable over time.

Data Tests

• Required columns exist.
• Data types are correct.
• Missing-value limits are respected.
• Unexpected categories are detected.

Model Tests

• Model loads successfully.
• Prediction schema is correct.
• Probability values remain between 0 and 1.
• Performance remains above agreed thresholds.

API Tests

• Valid requests return predictions.
• Invalid requests return useful errors.
• API response format remains stable.

• Optimizing only for accuracy.
• Ignoring class imbalance.
• Allowing data leakage.
• Using future information during training.
• Skipping a proper test set.
• Overfitting the model to historical data.
• Ignoring business costs of false positives and false negatives.
• Deploying without monitoring.
• Failing to version datasets and models.
• Treating correlation as proof of causation.

• ☐ Business problem clearly defined
• ☐ ML problem formulation documented
• ☐ Data sources identified
• ☐ Data quality checked
• ☐ Leakage risks reviewed
• ☐ Train/validation/test strategy defined
• ☐ Features documented
• ☐ Baseline model created
• ☐ Multiple models compared
• ☐ Appropriate metrics selected
• ☐ Model explainability considered
• ☐ Model serialized and versioned
• ☐ Prediction API tested
• ☐ Monitoring implemented
• ☐ Retraining process defined
• ☐ Business impact measured

This case study demonstrates how a real-world AI/ML solution moves through the complete lifecycle:

AI & Machine Learning Case Study

From Business Problem → Data → ML Model → API → Production → Monitoring