Advanced Pandas Features

85 min

Advanced Pandas features include multi-indexing, time series analysis, and custom functions, enabling sophisticated data manipulation beyond basic operations. These features handle complex data structures and enable advanced analysis patterns. Understanding advanced features enables solving complex data problems. Advanced features are essential for professional data science.

Multi-indexing (hierarchical indexing) enables working with data organized in multiple levels, like grouping by multiple dimensions. Multi-index DataFrames use tuples for indexing, enabling complex data organization. Use set_index() to create multi-index, loc[] to access data. Understanding multi-indexing enables complex data structures. Multi-indexing is powerful for hierarchical data.

Time series analysis uses Pandas' time series capabilities: date ranges (pd.date_range()), resampling (resample()), rolling windows (rolling()), and time-based indexing. Time series operations enable temporal analysis. Understanding time series enables analyzing time-based data. Time series analysis is essential for temporal data.

Custom functions with apply(), applymap(), and map() enable applying Python functions to DataFrames, Series, or elements. apply() works on Series/DataFrames, applymap() on elements, map() on Series. Understanding custom functions enables flexible transformations. Custom functions extend Pandas capabilities.

Performance optimization is important for large datasets—use vectorized operations instead of apply() when possible, use appropriate dtypes, use categorical types for repeated strings, and consider chunking for very large files. Understanding performance enables efficient processing. Performance is crucial for large datasets.

Best practices include using multi-indexing for hierarchical data, leveraging time series capabilities, using vectorized operations when possible, optimizing dtypes for memory, and understanding when to use custom functions. Understanding advanced features enables sophisticated data science. Advanced features are essential for complex problems.

Key Concepts

Advanced Pandas features include multi-indexing, time series, custom functions.
Multi-indexing enables hierarchical data organization.
Time series analysis enables temporal data operations.
Custom functions extend Pandas capabilities.
Performance optimization is important for large datasets.

Learning Objectives

Master

Using multi-indexing for hierarchical data
Performing time series analysis and operations
Applying custom functions with apply() and map()
Optimizing performance for large datasets

Develop

Understanding advanced data manipulation patterns
Designing efficient data processing workflows
Appreciating advanced features' role in data science

Tips

Use multi-indexing for hierarchical data: df.set_index(['col1', 'col2']).
Use resample() for time series aggregation: df.resample('M').sum().
Use vectorized operations instead of apply() when possible for performance.
Use categorical dtype for repeated strings to save memory.

Common Pitfalls

Using apply() when vectorized operations would work, losing performance.
Not understanding multi-indexing, causing indexing errors.
Not optimizing dtypes, wasting memory.
Not handling time zones in time series, causing errors.

Summary

Advanced Pandas features enable complex data transformations.
Multi-indexing, time series, and custom functions extend capabilities.
Performance optimization is important for large datasets.
Understanding advanced features enables sophisticated analysis.
Advanced features are essential for professional data science.

Exercise

Use advanced Pandas features including multi-indexing, time series, and custom functions.

import pandas as pd
import numpy as np

# Create a complex dataset with multi-index
np.random.seed(42)
dates = pd.date_range('2024-01-01', periods=90, freq='D')
stores = ['Store_A', 'Store_B', 'Store_C']
products = ['Laptop', 'Phone', 'Tablet']

# Create multi-index data
data = []
for date in dates:
    for store in stores:
        for product in products:
            data.append({
                'Date': date,
                'Store': store,
                'Product': product,
                'Sales': np.random.randint(10, 100),
                'Revenue': np.random.uniform(500, 2000),
                'Customers': np.random.randint(5, 25)
            })

df = pd.DataFrame(data)
print("Original data shape:", df.shape)
print("\nFirst few rows:")
print(df.head())

# 1. Multi-indexing
print("\n=== Multi-Indexing ===")
df_multi = df.set_index(['Date', 'Store', 'Product'])
print("Multi-index data:")
print(df_multi.head())

# Access data with multi-index
print("\nSales for Store_A, Laptop:")
print(df_multi.loc[('2024-01-01', 'Store_A', 'Laptop')])

# 2. Time series operations
print("\n=== Time Series Operations ===")
# Resample by month
monthly_sales = df.groupby('Store')['Sales'].resample('M', on='Date').sum()
print("Monthly sales by store:")
print(monthly_sales)

# Rolling statistics
rolling_avg = df.groupby('Store')['Sales'].rolling(window=7).mean()
print("\n7-day rolling average by store:")
print(rolling_avg.head(10))

# 3. Custom functions with apply
print("\n=== Custom Functions ===")
def categorize_revenue(revenue):
    if revenue < 1000:
        return 'Low'
    elif revenue < 1500:
        return 'Medium'
    else:
        return 'High'

df['Revenue_Category'] = df['Revenue'].apply(categorize_revenue)
print("Revenue categories:")
print(df['Revenue_Category'].value_counts())

# 4. Advanced grouping with custom aggregations
print("\n=== Advanced Grouping ===")
def custom_stats(group):
    return pd.Series({
        'Total_Sales': group['Sales'].sum(),
        'Avg_Revenue': group['Revenue'].mean(),
        'Max_Customers': group['Customers'].max(),
        'Sales_Count': len(group)
    })

store_stats = df.groupby('Store').apply(custom_stats)
print("Store statistics:")
print(store_stats)

# 5. Pivot with multiple aggregations
print("\n=== Pivot with Multiple Aggregations ===")
pivot_multi = df.pivot_table(
    values=['Sales', 'Revenue', 'Customers'],
    index='Store',
    columns='Product',
    aggfunc={
        'Sales': ['sum', 'mean'],
        'Revenue': ['sum', 'mean'],
        'Customers': 'sum'
    }
)
print("Multi-aggregation pivot table:")
print(pivot_multi)

# 6. Time-based grouping with custom periods
print("\n=== Time-based Grouping ===")
df['Month'] = df['Date'].dt.to_period('M')
df['Week'] = df['Date'].dt.to_period('W')

weekly_stats = df.groupby(['Store', 'Week'])['Sales'].sum().unstack(level=0)
print("Weekly sales by store:")
print(weekly_stats.head())

# 7. Performance optimization
print("\n=== Performance Optimization ===")
# Use categorical for repeated values
df['Store'] = df['Store'].astype('category')
df['Product'] = df['Product'].astype('category')

print("Memory usage before optimization:")
print(df.memory_usage(deep=True).sum())

# Use efficient data types
df['Sales'] = df['Sales'].astype('int16')
df['Customers'] = df['Customers'].astype('int8')

print("Memory usage after optimization:")
print(df.memory_usage(deep=True).sum())

Advanced Pandas Features