Exploratory Data Analysis with Pandas

Duration: 5 min

This module delves into the essential techniques and tools for performing Exploratory Data Analysis (EDA) using the Pandas library in Python. EDA is a crucial step in the data science pipeline, allowing you to understand the underlying patterns, distributions, and relationships within your dataset. By the end of this module, you will be proficient in using Pandas to load, manipulate, and visualize data, setting a solid foundation for more advanced data analysis tasks.

Loading and Inspecting Data

The first step in EDA is to load your dataset into a Pandas DataFrame and inspect its structure. Pandas provides powerful functions like read_csv to load data from various sources and head, info, and describe to get an initial understanding of the data.

import pandas as pd

# Load dataset
df = pd.read_csv('data.csv')

# Display the first 5 rows
print(df.head())

# Get a concise summary of the DataFrame
print(df.info())

# Generate descriptive statistics
print(df.describe())

Try it in Google Colab:

       A         B         C
0  0.469112 -0.282863 -1.509059
1 -0.282863 -1.509059 -1.135632
2 -1.509059 -1.135632  1.212112
3 -0.923060  2.565646 -0.424972
4  0.599605 -1.044236 -1.170299
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   A       5 non-null      float64
 1   B       5 non-null      float64
 2   C       5 non-null      float64
dtype: object
None
             A         B         C
count  5.000000  5.000000  5.000000
mean  -0.128549 -0.484679 -0.547303
std     0.899486  1.461975  1.167885
min   -1.509059 -1.509059 -1.509059
25%   -0.923060 -1.135632 -1.135632
50%   -0.282863 -0.424972 -0.424972
75%    0.469112  0.599605  0.599605
max    2.565646  2.565646  1.212112

Handling Missing Data

Missing data is a common issue in real-world datasets. Pandas offers several methods to handle missing values, such as dropna to remove them and fillna to replace them with specified values or methods like forward fill or backward fill.

import pandas as pd
import numpy as np

# Create a DataFrame with missing values
df = pd.DataFrame({'A': [1, 2, np.nan], 'B': [4, np.nan, 6], 'C': [7, 8, 9]})

# Remove rows with any missing values
df_cleaned = df.dropna()
print(df_cleaned)

# Fill missing values with a specified value
df_filled = df.fillna(0)
print(df_filled)

💡 Tip: When handling missing data, consider the nature of your dataset and the implications of each method. Removing data might lead to loss of information, while filling might introduce bias.

Key Concepts

Check Your Understanding