What is Data Analysis?
Data Analysis is the process of examining raw data to discover useful information, identify patterns, understand trends, and support better decisions.
Python is widely used for data analysis because it provides powerful libraries such as NumPy, Pandas, and Matplotlib.
📥 Data Collection
Collect data from CSV files, Excel files, databases, APIs, applications, and other sources.
🧹 Data Cleaning
Remove duplicates, handle missing values, fix incorrect data, and prepare the dataset.
🔎 Data Exploration
Examine rows, columns, statistics, distributions, and relationships within the data.
📊 Data Visualization
Create charts and graphs to communicate patterns and insights clearly.
A typical data-analysis project starts with raw data and ends with meaningful information that can support business, scientific, financial, or operational decisions.
- Easy-to-read syntax.
- Large ecosystem of data-analysis libraries.
- Works with CSV, Excel, JSON, SQL, and many other data sources.
- Powerful data-cleaning capabilities.
- Excellent visualization support.
- Can automate repetitive data-processing tasks.
- Can handle datasets ranging from small files to large workflows.
- Works well with machine learning and statistical tools.
| Library | Purpose |
|---|---|
| NumPy | Numerical calculations and arrays. |
| Pandas | Data manipulation and analysis. |
| Matplotlib | Charts and data visualization. |
| Seaborn | Statistical data visualization. |
| Scikit-learn | Machine learning and predictive modeling. |
Install the commonly used libraries using pip.
pip install pandas numpy matplotlibimport numpy as np
import pandas as pd
import matplotlib.pyplot as pltThese three libraries cover many common data-analysis tasks:
NumPy→ numerical operationsPandas→ data manipulationMatplotlib→ visualization
Before working with external files, we can create a small dataset directly in Python.
data = {
"Product": [
"Laptop",
"Mobile",
"Tablet",
"Monitor"
],
"Units": [
10,
25,
15,
12
],
"Price": [
60000,
25000,
18000,
22000
]
}
print(data){
'Product': ['Laptop', 'Mobile', 'Tablet', 'Monitor'],
'Units': [10, 25, 15, 12],
'Price': [60000, 25000, 18000, 22000]
}A Pandas DataFrame stores data in rows and columns, similar to a spreadsheet.
import pandas as pd
data = {
"Product": ["Laptop", "Mobile", "Tablet", "Monitor"],
"Units": [10, 25, 15, 12],
"Price": [60000, 25000, 18000, 22000]
}
df = pd.DataFrame(data)
print(df)| Product | Units | Price | |
|---|---|---|---|
| 0 | Laptop | 10 | 60000 |
| 1 | Mobile | 25 | 25000 |
| 2 | Tablet | 15 | 18000 |
| 3 | Monitor | 12 | 22000 |
The head() function is commonly used to
inspect the beginning of a dataset.
print(df.head())| Product | Units | Price | |
|---|---|---|---|
| 0 | Laptop | 10 | 60000 |
| 1 | Mobile | 25 | 25000 |
| 2 | Tablet | 15 | 18000 |
| 3 | Monitor | 12 | 22000 |
By default, head() displays the first five rows.
print(df.tail())
Use tail() to inspect the last rows of a dataset.
print(df.shape)df.shape returns (100, 5).
The info() function provides information
about columns, data types, and missing values.
df.info()
Use describe() to obtain common numerical
statistics.
print(df.describe())print(df["Product"])Selecting individual columns is one of the most common operations in data analysis.
selected = df[["Product", "Units"]]
print(selected)Filtering allows you to select rows that satisfy a condition.
result = df[df["Units"] > 12]
print(result)result = df[
(df["Units"] > 10) &
(df["Price"] > 20000)
]
print(result)& for AND and | for OR,
and place each condition inside parentheses.
sorted_df = df.sort_values(
by="Price",
ascending=False
)
print(sorted_df)Data analysis often involves creating new values from existing columns.
df["Total"] = df["Units"] * df["Price"]
print(df)Real-world datasets frequently contain missing values.
print(df.isnull().sum())This shows how many missing values exist in each column.
df["Price"] = df["Price"].fillna(0)
print(df)df = df.drop_duplicates()
print(df)CSV files are one of the most common data sources in practical data-analysis projects.
import pandas as pd
df = pd.read_csv("sales.csv")
print(df.head())import pandas as pd
df = pd.read_excel("sales.xlsx")
print(df.head())import pandas as pd
df = pd.read_json("customers.json")
print(df.head())
groupby() is useful when you need
summaries by category.
import pandas as pd
data = {
"Department": [
"IT",
"IT",
"HR",
"HR",
"Finance"
],
"Salary": [
70000,
80000,
50000,
55000,
65000
]
}
df = pd.DataFrame(data)
result = df.groupby("Department")["Salary"].mean()
print(result)| Department | Salary (Mean) |
|---|---|
| Finance | 65000.0 |
| HR | 52500.0 |
| IT | 75000.0 |
result = df.groupby("Department").agg(
Average_Salary=("Salary", "mean"),
Maximum_Salary=("Salary", "max"),
Minimum_Salary=("Salary", "min")
)
print(result)| Department | Average_Salary | Maximum_Salary | Minimum_Salary |
|---|---|---|---|
| Finance | 65000.0 | 65000 | 65000 |
| HR | 52500.0 | 55000 | 50000 |
| IT | 75000.0 | 80000 | 70000 |
Dates are important in sales, finance, website analytics, inventory, and many other datasets.
import pandas as pd
data = {
"Date": [
"2025-01-05",
"2025-02-10",
"2025-03-15"
],
"Sales": [
50000,
65000,
72000
]
}
df = pd.DataFrame(data)
df["Date"] = pd.to_datetime(df["Date"])
print(df)| Date | Sales | |
|---|---|---|
| 0 | 2025-01-05 | 50000 |
| 1 | 2025-02-10 | 65000 |
| 2 | 2025-03-15 | 72000 |
NumPy provides efficient numerical operations.
import numpy as np
sales = np.array([
100,
150,
200,
180,
220
])
print("Total:", np.sum(sales))
print("Average:", np.mean(sales))
print("Maximum:", np.max(sales))
print("Minimum:", np.min(sales))Total: 850
Average: 170.0
Maximum: 220
Minimum: 100After analyzing data, charts can be used to communicate important patterns.
import matplotlib.pyplot as plt
months = [
"Jan",
"Feb",
"Mar",
"Apr",
"May"
]
sales = [
100,
130,
150,
170,
210
]
plt.plot(
months,
sales,
marker="o"
)
plt.title("Monthly Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.grid(True)
plt.show()Consider an e-commerce company that stores order data. The data analyst needs to calculate revenue and identify the best-selling products.
| Product | Category | Units | Price |
|---|---|---|---|
| Laptop | Electronics | 20 | 60000 |
| Mobile | Electronics | 45 | 25000 |
| Chair | Furniture | 15 | 8000 |
| Desk | Furniture | 10 | 12000 |
import pandas as pd
import matplotlib.pyplot as plt
data = {
"Product": [
"Laptop",
"Mobile",
"Chair",
"Desk"
],
"Category": [
"Electronics",
"Electronics",
"Furniture",
"Furniture"
],
"Units": [
20,
45,
15,
10
],
"Price": [
60000,
25000,
8000,
12000
]
}
df = pd.DataFrame(data)
df["Revenue"] = df["Units"] * df["Price"]
print("Sales Data:")
print(df)
print("\nTotal Revenue:")
print(df["Revenue"].sum())
plt.bar(
df["Product"],
df["Revenue"]
)
plt.title("Product Revenue")
plt.xlabel("Product")
plt.ylabel("Revenue")
plt.show()| Product | Category | Units | Price | Revenue | |
|---|---|---|---|---|---|
| 0 | Laptop | Electronics | 20 | 60000 | 1200000 |
| 1 | Mobile | Electronics | 45 | 25000 | 1125000 |
| 2 | Chair | Furniture | 15 | 8000 | 120000 |
| 3 | Desk | Furniture | 10 | 12000 | 120000 |
An HR team can use Python to analyze employee salaries, departments, and performance.
import pandas as pd
data = {
"Name": [
"Amit",
"Priya",
"Rahul",
"Sneha",
"Vikas"
],
"Department": [
"IT",
"HR",
"IT",
"Finance",
"IT"
],
"Salary": [
70000,
55000,
85000,
65000,
90000
],
"Performance": [
82,
90,
88,
76,
95
]
}
df = pd.DataFrame(data)
print("Average Salary:")
print(df["Salary"].mean())
print("\nAverage Performance:")
print(df["Performance"].mean())
print("\nIT Employees:")
print(df[df["Department"] == "IT"])73000.0
Average Performance:
86.2
| Name | Department | Salary | Performance | |
|---|---|---|---|---|
| 0 | Amit | IT | 70000 | 82 |
| 2 | Rahul | IT | 85000 | 88 |
| 4 | Vikas | IT | 90000 | 95 |
A company can analyze customer purchase behavior.
import pandas as pd
data = {
"Customer": [
"A",
"B",
"C",
"D",
"E"
],
"Orders": [
5,
2,
8,
3,
10
],
"Amount": [
12000,
5000,
25000,
7000,
32000
]
}
df = pd.DataFrame(data)
high_value = df[df["Amount"] > 10000]
print("High Value Customers:")
print(high_value)| Customer | Orders | Amount | |
|---|---|---|---|
| 0 | A | 5 | 12000 |
| 2 | C | 8 | 25000 |
| 4 | E | 10 | 32000 |
import pandas as pd
import matplotlib.pyplot as plt
data = {
"Day": [
"Mon",
"Tue",
"Wed",
"Thu",
"Fri",
"Sat",
"Sun"
],
"Visitors": [
1200,
1500,
1800,
1700,
2200,
3000,
2800
]
}
df = pd.DataFrame(data)
print("Average Visitors:")
print(df["Visitors"].mean())
plt.plot(
df["Day"],
df["Visitors"],
marker="o"
)
plt.title("Website Traffic")
plt.xlabel("Day")
plt.ylabel("Visitors")
plt.grid(True)
plt.show()2028.5714285714286
| Day | Visitors | |
|---|---|---|
| 0 | Mon | 1200 |
| 1 | Tue | 1500 |
| 2 | Wed | 1800 |
| 3 | Thu | 1700 |
| 4 | Fri | 2200 |
| 5 | Sat | 3000 |
| 6 | Sun | 2800 |
Inventory analysis helps identify products that may require restocking.
import pandas as pd
data = {
"Product": [
"Laptop",
"Mobile",
"Keyboard",
"Mouse",
"Monitor"
],
"Stock": [
15,
50,
8,
30,
12
]
}
df = pd.DataFrame(data)
low_stock = df[df["Stock"] < 15]
print("Products Requiring Attention:")
print(low_stock)| Product | Stock | |
|---|---|---|
| 2 | Keyboard | 8 |
| 4 | Monitor | 12 |
This example combines data creation, calculation, filtering, grouping, statistics, and visualization.
import pandas as pd
import matplotlib.pyplot as plt
data = {
"Month": [
"Jan",
"Feb",
"Mar",
"Apr",
"May",
"Jun"
],
"Sales": [
120000,
145000,
160000,
150000,
190000,
220000
],
"Expenses": [
80000,
90000,
100000,
95000,
115000,
130000
]
}
df = pd.DataFrame(data)
# Calculate profit
df["Profit"] = df["Sales"] - df["Expenses"]
# Display data
print("Complete Dataset:")
print(df)
# Summary
print("\nTotal Sales:")
print(df["Sales"].sum())
print("\nTotal Expenses:")
print(df["Expenses"].sum())
print("\nTotal Profit:")
print(df["Profit"].sum())
print("\nAverage Monthly Sales:")
print(df["Sales"].mean())
# Visualization
plt.figure(figsize=(10, 5))
plt.plot(
df["Month"],
df["Sales"],
marker="o",
label="Sales"
)
plt.plot(
df["Month"],
df["Profit"],
marker="o",
label="Profit"
)
plt.title("Sales and Profit Analysis")
plt.xlabel("Month")
plt.ylabel("Amount")
plt.legend()
plt.grid(True)
plt.show()| Month | Sales | Expenses | Profit | |
|---|---|---|---|---|
| 0 | Jan | 120000 | 80000 | 40000 |
| 1 | Feb | 145000 | 90000 | 55000 |
| 2 | Mar | 160000 | 100000 | 60000 |
| 3 | Apr | 150000 | 95000 | 55000 |
| 4 | May | 190000 | 115000 | 75000 |
| 5 | Jun | 220000 | 130000 | 90000 |
985000
Total Expenses:
610000
Total Profit:
375000
Average Monthly Sales:
164166.66666666666
After analyzing data, the resulting DataFrame can be exported for reporting or further processing.
df.to_csv(
"sales_analysis.csv",
index=False
)df.to_excel(
"sales_analysis.xlsx",
index=False
)
Executing these methods writes the DataFrame directly to your working directory.
Setting index=False prevents Pandas from saving the row index numbers into your file.
Note: Exporting to Excel requires the optional openpyxl library to be installed.
| Operation | Python / Pandas Tool |
|---|---|
| Read CSV | pd.read_csv() |
| Read Excel | pd.read_excel() |
| View first rows | head() |
| View last rows | tail() |
| Dataset information | info() |
| Statistics | describe() |
| Missing values | isnull() |
| Remove duplicates | drop_duplicates() |
| Filter rows | Boolean conditions |
| Sort data | sort_values() |
| Group data | groupby() |
| Visualize data | Matplotlib |
| Data Analysis | Data Science |
|---|---|
| Focuses on understanding existing data. | Often includes predictive and machine-learning work. |
| Finds trends and patterns. | Builds predictive or statistical models. |
| Frequently uses Pandas and visualization. | May use Pandas, NumPy, Scikit-learn, and other tools. |
| Often supports reporting and business decisions. | Can support prediction, classification, recommendation, and automation. |
- Understand the business problem before analyzing data.
- Inspect the dataset before making assumptions.
- Check data types carefully.
- Identify missing values.
- Remove or investigate duplicate records.
- Check for invalid or unexpected values.
- Use appropriate filtering and aggregation techniques.
- Visualize important trends and relationships.
- Keep raw data separate from transformed data.
- Document important assumptions and calculations.
- Validate results before presenting conclusions.
- Choose visualizations that match the question being analyzed.
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load data
df = pd.read_csv("sales.csv")
# 2. Inspect data
print(df.head())
print(df.info())
# 3. Clean data
df = df.drop_duplicates()
# 4. Handle missing values
df = df.dropna()
# 5. Create calculated columns
df["Revenue"] = df["Units"] * df["Price"]
# 6. Analyze
print(df.describe())
# 7. Group data
summary = df.groupby("Product")["Revenue"].sum()
print(summary)
# 8. Visualize
summary.plot(kind="bar")
plt.title("Revenue by Product")
plt.xlabel("Product")
plt.ylabel("Revenue")
plt.tight_layout()
plt.show()Keyboard 64000
Laptop 1200000
Mobile 1125000
Monitor 144000
Mouse 300000
Name: Revenue, dtype: int64
Workflow Summary: This end-to-end blueprint ties together data ingestion, cleaning, feature engineering, aggregation, and plotting to ensure robust and reproducible data analysis conclusions.
- Start with a clearly defined analysis question.
- Understand where the data comes from.
- Inspect rows, columns, and data types.
- Check for missing and duplicate data.
- Clean the data before drawing conclusions.
- Use Pandas for tabular data manipulation.
- Use NumPy for numerical operations.
- Use Matplotlib for data visualization.
- Use filtering to answer specific questions.
- Use grouping and aggregation for summaries.
- Use charts to communicate important patterns.
- Validate calculations before reporting results.
- Keep the original raw data unchanged when possible.
- Document important transformations and assumptions.
- Interpret results in the context of the original business problem.
Python Data Analysis
Turn raw data into useful information with Python.