Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Pandas 1.x Cookbook
  • Table Of Contents Toc
  • Feedback & Rating feedback
Pandas 1.x Cookbook

Pandas 1.x Cookbook

By : Matthew Harrison, Theodore Petrou
4.5 (28)
Buy this Book
close
close
Pandas 1.x Cookbook

Pandas 1.x Cookbook

4.5 (28)
By: Matthew Harrison, Theodore Petrou
Buy this Book

Overview of this book

The pandas library is massive, and it's common for frequent users to be unaware of many of its more impressive features. The official pandas documentation, while thorough, does not contain many useful examples of how to piece together multiple commands as one would do during an actual analysis. This book guides you, as if you were looking over the shoulder of an expert, through situations that you are highly likely to encounter. This new updated and revised edition provides you with unique, idiomatic, and fun recipes for both fundamental and advanced data manipulation tasks with pandas. Some recipes focus on achieving a deeper understanding of basic principles, or comparing and contrasting two similar operations. Other recipes will dive deep into a particular dataset, uncovering new and unexpected insights along the way. Many advanced recipes combine several different features across the pandas library to generate results.
Table of Contents (17 chapters)
close
close
close
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Pandas Foundations
Importing pandas
Introduction
The pandas DataFrame
DataFrame attributes
Understanding data types
Selecting a column
Calling Series methods
Series operations
Chaining Series methods
Renaming column names
Creating and deleting columns
Free Chapter
2
Essential DataFrame Operations
Introduction
Selecting multiple DataFrame columns
Selecting columns with methods
Ordering column names
Summarizing a DataFrame
Chaining DataFrame methods
DataFrame operations
Comparing missing values
Transposing the direction of a DataFrame operation
Determining college campus diversity
3
Creating and Persisting DataFrames
Introduction
Creating DataFrames from scratch
Writing CSV
Reading large CSV files
Using Excel files
Working with ZIP files
Working with databases
Reading JSON
Reading HTML tables
4
Beginning Data Analysis
chevron up
Introduction
Developing a data analysis routine
Data dictionaries
Reducing memory by changing data types
Selecting the smallest of the largest
Selecting the largest of each group by sorting
Replicating nlargest with sort_values
Calculating a trailing stop order price
5
Exploratory Data Analysis
Introduction
Summary statistics
Column types
Categorical data
Continuous data
Comparing continuous values across categories
Comparing two continuous columns
Comparing categorical values with categorical values
Using the pandas profiling library
6
Selecting Subsets of Data
Introduction
Selecting Series data
Selecting DataFrame rows
Selecting DataFrame rows and columns simultaneously
Selecting data with both integers and labels
Slicing lexicographically
7
Filtering Rows
Introduction
Calculating Boolean statistics
Constructing multiple Boolean conditions
Filtering with Boolean arrays
Comparing row filtering and index filtering
Selecting with unique and sorted indexes
Translating SQL WHERE clauses
Improving the readability of Boolean indexing with the query method
Preserving Series size with the .where method
Masking DataFrame rows
Selecting with Booleans, integer location, and labels
8
Index Alignment
Introduction
Examining the Index object
Producing Cartesian products
Exploding indexes
Filling values with unequal indexes
Adding columns from different DataFrames
Highlighting the maximum value from each column
Replicating idxmax with method chaining
Finding the most common maximum of columns
9
Grouping for Aggregation, Filtration, and Transformation
Introduction
Defining an aggregation
Grouping and aggregating with multiple columns and functions
Removing the MultiIndex after grouping
Grouping with a custom aggregation function
Customizing aggregating functions with *args and **kwargs
Examining the groupby object
Filtering for states with a minority majority
Transforming through a weight loss bet
Calculating weighted mean SAT scores per state with apply
Grouping by continuous variables
Counting the total number of flights between cities
Finding the longest streak of on-time flights
10
Restructuring Data into a Tidy Form
Introduction
Tidying variable values as column names with stack
Tidying variable values as column names with melt
Stacking multiple groups of variables simultaneously
Inverting stacked data
Unstacking after a groupby aggregation
Replicating pivot_table with a groupby aggregation
Renaming axis levels for easy reshaping
Tidying when multiple variables are stored as column names
Tidying when multiple variables are stored as a single column
Tidying when two or more values are stored in the same cell
Tidying when variables are stored in column names and values
11
Combining Pandas Objects
Introduction
Appending new rows to DataFrames
Concatenating multiple DataFrames together
Understanding the differences between concat, join, and merge
Connecting to SQL databases
12
Time Series Analysis
Introduction
Understanding the difference between Python and pandas date tools
Slicing time series intelligently
Filtering columns with time data
Using methods that only work with a DatetimeIndex
Counting the number of weekly crimes
Aggregating weekly crime and traffic accidents separately
Measuring crime by weekday and year
Grouping with anonymous functions with a DatetimeIndex
Grouping by a Timestamp and another column
13
Visualization with Matplotlib, Pandas, and Seaborn
Introduction
Getting started with matplotlib
Object-oriented guide to matplotlib
Visualizing data with matplotlib
Plotting basics with pandas
Visualizing the flights dataset
Stacking area charts to discover emerging trends
Understanding the differences between seaborn and pandas
Multivariate analysis with seaborn Grids
Uncovering Simpson's Paradox in the diamonds dataset with seaborn
14
Debugging and Testing Pandas
Code to transform data
Apply performance
Improving apply performance with Dask, Pandarell, Swifter, and more
Inspecting code
Debugging in Jupyter
Managing data integrity with Great Expectations
Using pytest with pandas
Generating tests with Hypothesis
15
Other Books You May Enjoy
Other Books You May Enjoy
16
Index
Index

Data dictionaries

A crucial part of data analysis involves creating and maintaining a data dictionary. A data dictionary is a table of metadata and notes on each column of data. One of the primary purposes of a data dictionary is to explain the meaning of the column names. The college dataset uses a lot of abbreviations that are likely to be unfamiliar to an analyst who is inspecting it for the first time.

A data dictionary for the college dataset is provided in the following college_data_dictionary.csv file:

>>> pd.read_csv("data/college_data_dictionary.csv")
    column_name  description
0        INSTNM  Institut...
1          CITY  City Loc...
2        STABBR  State Ab...
3          HBCU  Historic...
4       MENONLY  0/1 Men ...
..          ...          ...
22      PCTPELL  Percent ...
23     PCTFLOAN  Percent ...
24      UG25ABV  Percent ...
25  MD_EARN_...  Median E...
26  GRAD_DEB...  Median d...

As you can see, it is immensely helpful in deciphering...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 3
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY