Data Analysis and Aggregation

80 min

Pandas provides powerful tools for data analysis and aggregation, enabling you to summarize, group, and analyze data efficiently. Analysis operations include grouping, aggregating, pivoting, and computing statistics. These operations are optimized and work on large datasets. Understanding analysis tools enables extracting insights from data. Data analysis is the core of data science.

GroupBy operations allow you to analyze data by categories, splitting data into groups, applying functions to each group, and combining results. GroupBy is one of Pandas' most powerful features, enabling category-based analysis. Use df.groupby() to group by columns, then apply aggregation functions (sum, mean, count, etc.). Understanding GroupBy enables sophisticated data analysis. GroupBy is essential for data analysis.

Aggregation functions (sum, mean, count, min, max, std, etc.) compute statistics on groups or entire DataFrames. Aggregations can be applied to specific columns or all numeric columns. Multiple aggregations can be applied simultaneously. Understanding aggregations enables data summarization. Aggregations are fundamental to analysis.

Pivot tables and cross-tabulations help summarize complex data by reorganizing and aggregating data. pd.pivot_table() creates pivot tables with rows, columns, values, and aggregation functions. pd.crosstab() creates frequency tables. Understanding pivot tables enables data reorganization. Pivot tables are powerful for data summarization.

Multi-level grouping enables grouping by multiple columns, creating hierarchical analysis. Multi-level grouping provides deeper insights into data relationships. Results can be unstacked or stacked for different views. Understanding multi-level grouping enables complex analysis. Multi-level grouping is essential for sophisticated analysis.

Best practices include using GroupBy for category-based analysis, applying appropriate aggregation functions, using pivot tables for data reorganization, understanding multi-level grouping, and documenting analysis steps. Understanding data analysis and aggregation enables extracting insights. Analysis is essential for data science.

Key Concepts

Pandas provides powerful tools for data analysis and aggregation.
GroupBy operations allow analysis by categories.
Aggregation functions compute statistics (sum, mean, count, etc.).
Pivot tables and cross-tabulations summarize complex data.
Multi-level grouping enables hierarchical analysis.

Learning Objectives

Master

Using GroupBy operations for category-based analysis
Applying aggregation functions to groups and DataFrames
Creating pivot tables and cross-tabulations
Performing multi-level grouping and analysis

Develop

Understanding data analysis patterns
Designing effective analysis workflows
Appreciating analysis tools' role in data science

Tips

Use df.groupby() for category-based analysis.
Apply multiple aggregations: df.groupby('col').agg(['mean', 'sum', 'count']).
Use pd.pivot_table() for data reorganization.
Use pd.crosstab() for frequency tables.

Common Pitfalls

Not understanding GroupBy mechanics, getting unexpected results.
Applying wrong aggregation functions, getting incorrect summaries.
Not handling missing values in groups, causing errors.
Creating overly complex pivot tables, losing clarity.

Summary

Pandas provides powerful tools for data analysis and aggregation.
GroupBy operations enable category-based analysis.
Aggregation functions compute statistics on groups.
Pivot tables and cross-tabulations summarize complex data.
Understanding analysis tools enables extracting insights.

Exercise

Perform comprehensive data analysis using groupby, pivot tables, and aggregations.

import pandas as pd
import numpy as np

# Create a comprehensive sales dataset
np.random.seed(42)
dates = pd.date_range('2024-01-01', periods=365, freq='D')
products = ['Laptop', 'Phone', 'Tablet', 'Monitor', 'Keyboard']
regions = ['North', 'South', 'East', 'West']

sales_data = []
for date in dates:
    for _ in range(np.random.randint(5, 15)):  # 5-15 sales per day
        sale = {
            'Date': date,
            'Product': np.random.choice(products),
            'Region': np.random.choice(regions),
            'Quantity': np.random.randint(1, 5),
            'Unit_Price': np.random.uniform(100, 2000),
            'Customer_ID': np.random.randint(1000, 9999)
        }
        sales_data.append(sale)

df = pd.DataFrame(sales_data)
df['Total_Revenue'] = df['Quantity'] * df['Unit_Price']

print("Sales data shape:", df.shape)
print("\nFirst few rows:")
print(df.head())

# 1. Basic aggregations
print("\n=== Basic Aggregations ===")
print("Total revenue:", df['Total_Revenue'].sum())
print("Average unit price:", df['Unit_Price'].mean())
print("Total quantity sold:", df['Quantity'].sum())

# 2. GroupBy analysis
print("\n=== GroupBy Analysis ===")
# Revenue by product
product_revenue = df.groupby('Product')['Total_Revenue'].agg(['sum', 'mean', 'count'])
print("Revenue by product:")
print(product_revenue)

# Revenue by region
region_revenue = df.groupby('Region')['Total_Revenue'].agg(['sum', 'mean', 'count'])
print("\nRevenue by region:")
print(region_revenue)

# 3. Multi-level grouping
print("\n=== Multi-level Grouping ===")
product_region = df.groupby(['Product', 'Region'])['Total_Revenue'].sum().unstack()
print("Revenue by product and region:")
print(product_region)

# 4. Time-based analysis
print("\n=== Time-based Analysis ===")
monthly_revenue = df.groupby(df['Date'].dt.to_period('M'))['Total_Revenue'].sum()
print("Monthly revenue:")
print(monthly_revenue.head())

# 5. Pivot table
print("\n=== Pivot Table ===")
pivot_table = df.pivot_table(
    values='Total_Revenue',
    index='Product',
    columns='Region',
    aggfunc='sum',
    fill_value=0
)
print("Revenue pivot table:")
print(pivot_table)

# 6. Cross-tabulation
print("\n=== Cross-tabulation ===")
cross_tab = pd.crosstab(df['Product'], df['Region'], values=df['Quantity'], aggfunc='sum')
print("Quantity sold by product and region:")
print(cross_tab)

Data Analysis and Aggregation