Exploratory Data Analysis Using Python

What is Exploratory Data Analysis?

Exploratory Data Analysis, commonly called EDA, is the process of investigating a dataset before building reports, statistical models, or machine learning solutions. EDA helps discover patterns, missing values, unusual records, relationships, distributions, and potential data-quality problems.

Load Data

→

Inspect

→

Clean

→

Analyze

→

Visualize

→

Find Insights

Data Inspection

Understand rows, columns, data types, missing values, duplicates, and basic structure.

Statistical Analysis

Calculate counts, averages, minimums, maximums, percentiles, and other summary statistics.

Visualization

Use charts to identify trends, distributions, relationships, and unusual observations.

Business Insights

Convert patterns found in the data into useful questions, explanations, and decisions.

1. Installing Required Libraries

EDA commonly uses Pandas, NumPy, Matplotlib, and Seaborn.

Install Libraries

pip install pandas numpy matplotlib seaborn

Tip: Many Python data-science environments already include these libraries.

2. Importing EDA Libraries

Python

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

These libraries cover most of the basic tasks required during exploratory analysis.

3. Creating a Real-Time Sales Dataset

Suppose an e-commerce company wants to analyze its sales data. We will create a small example dataset containing customer, product, city, quantity, price, and rating information.

Python

data = {
    "Customer": [
        "Amit", "Priya", "Rahul", "Sneha",
        "Vikas", "Neha", "Arjun", "Pooja",
        "Karan", "Riya"
    ],
    "Product": [
        "Laptop", "Phone", "Tablet", "Laptop",
        "Phone", "Monitor", "Phone", "Tablet",
        "Laptop", "Monitor"
    ],
    "City": [
        "Pune", "Mumbai", "Delhi", "Pune",
        "Bangalore", "Mumbai", "Delhi", "Pune",
        "Bangalore", "Delhi"
    ],
    "Quantity": [
        1, 2, 3, 1, 4,
        2, 2, 3, 1, 2
    ],
    "Price": [
        60000, 30000, 25000, 60000, 30000,
        20000, 30000, 25000, 60000, 20000
    ],
    "Rating": [
        5, 4, 4, 5, 3,
        4, 5, 4, 5, 3
    ]
}

df = pd.DataFrame(data)

print(df)

Real-Time Scenario: This represents the type of transactional data an e-commerce analyst may receive from an order system.

4. Understanding the Dataset

The first step in EDA is understanding what the dataset contains.

Python

print(df.shape)
print(df.columns)
print(df.dtypes)

Important: shape tells you the number of rows and columns, columns shows column names, and dtypes shows the data types.

5. Viewing the First Records

Use head() to quickly inspect the beginning of a dataset.

Python

print(df.head())

You can also specify the number of rows:

Python

print(df.head(3))

6. Viewing the Last Records

Python

print(df.tail())

tail() is useful for checking the end of a dataset and verifying that records loaded correctly.

7. Getting Dataset Information

Python

df.info()

info() provides a compact overview of the DataFrame, including non-null values and data types.

8. Statistical Summary with describe()

Python

print(df.describe())

Common statistics: count, mean, standard deviation, minimum, quartiles, and maximum.

9. Statistical Summary of Text Columns

Python

print(
    df.describe(include="object")
)

This can summarize categorical columns such as Product, City, and Customer.

10. Checking Missing Values

Missing values can affect analysis and should be identified early.

Python

print(df.isnull().sum())

Why? Missing values may change averages, counts, charts, and downstream statistical analysis.

11. Checking Missing Values as Percentages

Python

missing_percentage = (
    df.isnull().sum()
    / len(df)
) * 100

print(missing_percentage)

Percentages make it easier to understand how significant missing data is for each column.

12. Checking Duplicate Rows

Python

print(df.duplicated().sum())

Duplicate records can cause counts, totals, and averages to be incorrect.

13. Checking Unique Values

Unique values help you understand categorical variables.

Python

print(df["Product"].unique())
print(df["City"].unique())

14. Counting Categories

Python

print(
    df["Product"].value_counts()
)

value_counts() is useful for identifying the most common categories.

15. Analyzing a Single Column

Python

print(df["Price"].mean())
print(df["Price"].median())
print(df["Price"].min())
print(df["Price"].max())

EDA Question: What is the typical product price, and how wide is the range of prices?

16. Calculating Sales Revenue

In this example, revenue can be calculated from quantity multiplied by price.

Python

df["Revenue"] = (
    df["Quantity"] *
    df["Price"]
)

print(df[[
    "Product",
    "Quantity",
    "Price",
    "Revenue"
]])

17. Finding Total Revenue

Python

total_revenue = df["Revenue"].sum()

print("Total Revenue:", total_revenue)

18. Grouping Data by Product

Grouping helps answer business questions such as: Which product generates the most revenue?

Python

product_sales = (
    df.groupby("Product")["Revenue"]
    .sum()
    .sort_values(ascending=False)
)

print(product_sales)

19. Grouping Data by City

Python

city_sales = (
    df.groupby("City")["Revenue"]
    .sum()
    .sort_values(ascending=False)
)

print(city_sales)

Business Use: Regional teams can compare revenue generated by different cities.

20. Grouping with Multiple Aggregations

Python

summary = df.groupby("Product").agg({
    "Quantity": "sum",
    "Price": ["mean", "min", "max"],
    "Revenue": "sum",
    "Rating": "mean"
})

print(summary)

Multiple aggregations provide a broader view of each category.

21. Sorting Data During EDA

Python

highest_revenue = df.sort_values(
    "Revenue",
    ascending=False
)

print(
    highest_revenue[
        ["Customer", "Product", "Revenue"]
    ]
)

22. Finding the Top Customers

Customer-level aggregation can reveal which customers generated the highest transaction values.

Python

top_customers = (
    df.groupby("Customer")["Revenue"]
    .sum()
    .sort_values(ascending=False)
    .head(5)
)

print(top_customers)

23. Filtering Data During EDA

Filtering lets you investigate specific parts of a dataset.

Python

high_value_orders = df[
    df["Revenue"] > 50000
]

print(high_value_orders)

24. Analyzing Numeric Distributions

Histograms help identify the shape and spread of numerical data.

Python

plt.figure(figsize=(9, 5))

plt.hist(
    df["Price"],
    bins=5
)

plt.title("Product Price Distribution")
plt.xlabel("Price")
plt.ylabel("Frequency")

plt.show()

25. Box Plot for Detecting Outliers

Python

plt.figure(figsize=(8, 5))

sns.boxplot(
    y=df["Revenue"]
)

plt.title("Revenue Distribution")

plt.show()

Important: A point that appears unusual is not automatically an error. It should be investigated using business context.

26. Analyzing Relationships Between Variables

Scatter plots can help investigate whether two numerical variables appear to have a relationship.

Python

plt.figure(figsize=(8, 5))

plt.scatter(
    df["Quantity"],
    df["Revenue"]
)

plt.title("Quantity vs Revenue")
plt.xlabel("Quantity")
plt.ylabel("Revenue")

plt.show()

27. Correlation Analysis

Correlation measures the strength and direction of a linear relationship between numerical variables.

Python

numeric_columns = [
    "Quantity",
    "Price",
    "Rating",
    "Revenue"
]

correlation = df[
    numeric_columns
].corr()

print(correlation)

Remember: Correlation does not by itself establish causation.

28. Visualizing Correlation with a Heatmap

Python

plt.figure(figsize=(8, 6))

sns.heatmap(
    correlation,
    annot=True
)

plt.title("Correlation Heatmap")

plt.show()

29. Analyzing Categorical Variables

Bar charts are useful for understanding category frequency.

Python

plt.figure(figsize=(8, 5))

sns.countplot(
    data=df,
    x="Product"
)

plt.title("Number of Orders by Product")
plt.xlabel("Product")
plt.ylabel("Number of Orders")

plt.xticks(rotation=20)

plt.show()

30. Product Revenue Visualization

Python

product_revenue = (
    df.groupby("Product")["Revenue"]
    .sum()
    .reset_index()
)

plt.figure(figsize=(9, 5))

sns.barplot(
    data=product_revenue,
    x="Product",
    y="Revenue"
)

plt.title("Revenue by Product")
plt.xlabel("Product")
plt.ylabel("Revenue")

plt.xticks(rotation=20)

plt.show()

31. City Revenue Visualization

Python

city_revenue = (
    df.groupby("City")["Revenue"]
    .sum()
    .reset_index()
)

plt.figure(figsize=(9, 5))

sns.barplot(
    data=city_revenue,
    x="City",
    y="Revenue"
)

plt.title("Revenue by City")
plt.xlabel("City")
plt.ylabel("Revenue")

plt.show()

32. Analyzing Average Rating

Python

rating_summary = (
    df.groupby("Product")["Rating"]
    .mean()
    .sort_values(ascending=False)
)

print(rating_summary)

Average ratings can be examined alongside sales and revenue to understand whether customer feedback differs by product.

33. Finding Minimum and Maximum Values

Python

print("Minimum Price:", df["Price"].min())
print("Maximum Price:", df["Price"].max())

print("Minimum Revenue:", df["Revenue"].min())
print("Maximum Revenue:", df["Revenue"].max())

34. Finding Percentiles

Percentiles help understand where observations fall within a numerical distribution.

Python

print(
    df["Revenue"].quantile(0.25)
)

print(
    df["Revenue"].quantile(0.50)
)

print(
    df["Revenue"].quantile(0.75)
)

35. Detecting Outliers with IQR

The Interquartile Range method is commonly used to identify values that fall unusually far from the central range.

Python

Q1 = df["Revenue"].quantile(0.25)
Q3 = df["Revenue"].quantile(0.75)

IQR = Q3 - Q1

lower_limit = Q1 - 1.5 * IQR
upper_limit = Q3 + 1.5 * IQR

outliers = df[
    (df["Revenue"] < lower_limit) |
    (df["Revenue"] > upper_limit)
]

print(outliers)

36. Real-Time Example: Customer Purchase Analysis

An e-commerce company wants to understand customer purchasing behavior.

Python

customer_summary = (
    df.groupby("Customer")
    .agg(
        Orders=("Product", "count"),
        TotalQuantity=("Quantity", "sum"),
        TotalRevenue=("Revenue", "sum"),
        AverageRating=("Rating", "mean")
    )
    .sort_values(
        "TotalRevenue",
        ascending=False
    )
)

print(customer_summary)

Analysis: This creates a customer-level summary containing order count, quantity, revenue, and average rating.

37. Real-Time Example: Product Performance Analysis

Python

product_analysis = (
    df.groupby("Product")
    .agg(
        Orders=("Product", "count"),
        QuantitySold=("Quantity", "sum"),
        AveragePrice=("Price", "mean"),
        Revenue=("Revenue", "sum"),
        AverageRating=("Rating", "mean")
    )
    .sort_values(
        "Revenue",
        ascending=False
    )
)

print(product_analysis)

Business Question: Which products generate the most revenue, and how do their sales volume, prices, and ratings compare?

38. Real-Time Example: City Performance Analysis

Python

city_analysis = (
    df.groupby("City")
    .agg(
        Orders=("Product", "count"),
        QuantitySold=("Quantity", "sum"),
        Revenue=("Revenue", "sum"),
        AverageRating=("Rating", "mean")
    )
    .sort_values(
        "Revenue",
        ascending=False
    )
)

print(city_analysis)

39. Crosstab Analysis

A crosstab can summarize the relationship between categorical variables.

Python

city_product = pd.crosstab(
    df["City"],
    df["Product"]
)

print(city_product)

This can show how frequently different products appear across cities.

40. Pivot Table for EDA

Pivot tables are useful for multi-dimensional summaries.

Python

pivot = pd.pivot_table(
    df,
    values="Revenue",
    index="City",
    columns="Product",
    aggfunc="sum",
    fill_value=0
)

print(pivot)

41. Visualizing a Pivot Table

Python

plt.figure(figsize=(10, 6))

sns.heatmap(
    pivot,
    annot=True,
    fmt=".0f"
)

plt.title(
    "Revenue by City and Product"
)

plt.show()

42. Analyzing Data by Multiple Conditions

EDA often requires investigating combinations of variables.

Python

result = df[
    (df["City"] == "Pune") &
    (df["Revenue"] > 50000)
]

print(result)

Example Question: Which high-value transactions occurred in Pune?

43. Real-Time Example: Finding High-Value Orders

A business analyst wants to identify transactions above a chosen revenue threshold.

Python

threshold = 50000

high_value_orders = df[
    df["Revenue"] >= threshold
].sort_values(
    "Revenue",
    ascending=False
)

print(
    high_value_orders[
        [
            "Customer",
            "Product",
            "City",
            "Revenue"
        ]
    ]
)

44. Checking Data Quality During EDA

Check	Python Method	Why It Matters
Rows and columns	`df.shape`	Understand dataset size
Data types	`df.dtypes`	Verify column types
Missing values	`df.isnull().sum()`	Find incomplete records
Duplicates	`df.duplicated().sum()`	Detect repeated records
Unique categories	`df["column"].unique()`	Find unexpected categories
Statistics	`df.describe()`	Understand numerical distributions

45. EDA Summary Metrics

Rows

Dataset Size

Nulls

Missing Values

Stats

Numerical Summary

Insights

Patterns Discovered

In a real project, these summary metrics can be generated automatically from the dataset.

46. Complete Real-Time EDA Example

The following example demonstrates a complete exploratory workflow for an e-commerce sales dataset.

Complete EDA Project

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# --------------------------------
# 1. Create the dataset
# --------------------------------

df = pd.DataFrame({
    "Customer": [
        "Amit", "Priya", "Rahul", "Sneha",
        "Vikas", "Neha", "Arjun", "Pooja",
        "Karan", "Riya", "Amit", "Priya"
    ],

    "Product": [
        "Laptop", "Phone", "Tablet", "Laptop",
        "Phone", "Monitor", "Phone", "Tablet",
        "Laptop", "Monitor", "Phone", "Laptop"
    ],

    "City": [
        "Pune", "Mumbai", "Delhi", "Pune",
        "Bangalore", "Mumbai", "Delhi", "Pune",
        "Bangalore", "Delhi", "Pune", "Mumbai"
    ],

    "Quantity": [
        1, 2, 3, 1,
        4, 2, 2, 3,
        1, 2, 3, 1
    ],

    "Price": [
        60000, 30000, 25000, 60000,
        30000, 20000, 30000, 25000,
        60000, 20000, 30000, 60000
    ],

    "Rating": [
        5, 4, 4, 5,
        3, 4, 5, 4,
        5, 3, 4, 5
    ]
})

# --------------------------------
# 2. Inspect the dataset
# --------------------------------

print("Shape:")
print(df.shape)

print("\nColumns:")
print(df.columns)

print("\nData Types:")
print(df.dtypes)

print("\nFirst Records:")
print(df.head())

# --------------------------------
# 3. Check missing values
# --------------------------------

print("\nMissing Values:")
print(df.isnull().sum())

# --------------------------------
# 4. Check duplicates
# --------------------------------

print("\nDuplicate Rows:")
print(df.duplicated().sum())

# --------------------------------
# 5. Create Revenue
# --------------------------------

df["Revenue"] = (
    df["Quantity"] *
    df["Price"]
)

# --------------------------------
# 6. Statistical summary
# --------------------------------

print("\nStatistics:")
print(df.describe())

# --------------------------------
# 7. Product analysis
# --------------------------------

product_summary = (
    df.groupby("Product")
    .agg(
        Orders=("Product", "count"),
        Quantity=("Quantity", "sum"),
        Revenue=("Revenue", "sum"),
        AverageRating=("Rating", "mean")
    )
    .sort_values(
        "Revenue",
        ascending=False
    )
)

print("\nProduct Summary:")
print(product_summary)

# --------------------------------
# 8. City analysis
# --------------------------------

city_summary = (
    df.groupby("City")
    .agg(
        Orders=("Product", "count"),
        Revenue=("Revenue", "sum"),
        Quantity=("Quantity", "sum")
    )
    .sort_values(
        "Revenue",
        ascending=False
    )
)

print("\nCity Summary:")
print(city_summary)

# --------------------------------
# 9. Correlation
# --------------------------------

numeric_columns = [
    "Quantity",
    "Price",
    "Rating",
    "Revenue"
]

correlation = df[
    numeric_columns
].corr()

print("\nCorrelation:")
print(correlation)

# --------------------------------
# 10. Visualization dashboard
# --------------------------------

fig, axes = plt.subplots(
    2,
    2,
    figsize=(14, 9)
)

# Product revenue
sns.barplot(
    data=product_summary.reset_index(),
    x="Product",
    y="Revenue",
    ax=axes[0, 0]
)

axes[0, 0].set_title(
    "Revenue by Product"
)

axes[0, 0].tick_params(
    axis="x",
    rotation=20
)

# City revenue
sns.barplot(
    data=city_summary.reset_index(),
    x="City",
    y="Revenue",
    ax=axes[0, 1]
)

axes[0, 1].set_title(
    "Revenue by City"
)

axes[0, 1].tick_params(
    axis="x",
    rotation=20
)

# Revenue distribution
axes[1, 0].hist(
    df["Revenue"],
    bins=6
)

axes[1, 0].set_title(
    "Revenue Distribution"
)

axes[1, 0].set_xlabel(
    "Revenue"
)

axes[1, 0].set_ylabel(
    "Frequency"
)

# Correlation heatmap
sns.heatmap(
    correlation,
    annot=True,
    ax=axes[1, 1]
)

axes[1, 1].set_title(
    "Correlation Heatmap"
)

plt.tight_layout()

plt.show()

Complete Workflow: Load → Inspect → Validate → Create Metrics → Summarize → Group → Visualize → Interpret.

47. EDA Questions for a Real Business Dataset

Data Quality

How many rows are present?
Are there missing values?
Are there duplicate records?
Are data types correct?

Customers

Who are the highest-value customers?
How many orders does each customer place?
What is the average order value?

Products

Which products sell the most?
Which products generate the most revenue?
Which products have unusual ratings?

Locations

Which cities generate the most revenue?
Which cities have the most orders?
Are there regional differences?

Numerical Data

What are the minimum and maximum values?
What are the averages and medians?
Are there unusual observations?

Relationships

Are numerical variables correlated?
Do charts reveal useful patterns?
What requires further investigation?

48. EDA Best Practices

Understand the business problem before analyzing the data.
Inspect the structure of the dataset first.
Check missing values and duplicates.
Verify data types and category values.
Use descriptive statistics to understand numerical columns.
Use grouping and aggregation to answer business questions.
Visualize distributions before making assumptions.
Investigate unusual values instead of automatically deleting them.
Use correlation as an exploratory measure, not proof of causation.
Compare numerical analysis with visual evidence.
Document important assumptions and transformations.
Keep EDA reproducible by writing Python code instead of relying only on manual inspection.

49. EDA vs Data Cleaning vs Data Visualization

Activity	Main Purpose	Typical Python Tools
Data Cleaning	Fix or prepare data-quality problems	Pandas
EDA	Understand patterns, distributions, relationships, and anomalies	Pandas, NumPy, Matplotlib, Seaborn
Data Visualization	Communicate patterns through charts and graphs	Matplotlib, Seaborn, Pandas

50. Important EDA Rules

Always inspect the dataset before analyzing it.
Check the number of rows and columns.
Understand every important column and its data type.
Check missing values.
Check duplicate records.
Inspect unique categorical values.
Use describe() for numerical summaries.
Use grouping and aggregation to answer business questions.
Use histograms and box plots to study distributions.
Use scatter plots to investigate relationships.
Use correlation carefully and do not interpret it as causation.
Investigate outliers using business context.
Use heatmaps and dashboards to summarize important patterns.
Document important findings and assumptions.
EDA should lead to better questions and more informed analysis.

Key Idea: Exploratory Data Analysis is the bridge between raw data and meaningful understanding. Before building a model or presenting a business report, use EDA to understand what the data actually contains.

Python Exploratory Data Analysis • Real-Time Examples • Data Analysis Learning