Optimizing Pandas Memory Usage for Baseball Game Data

Generally, pandas handles datasets smaller than 100 MB without performance issues, but processing data from 100 MB to several gigabytes can become slow and may cause out‑of‑memory failures.

The example uses a 130‑year Major League Baseball dataset sourced from Retrosheet, originally stored in 127 CSV files and loaded with csvkit .

Calling DataFrame.info(memory_usage='deep') reveals the DataFrame has 171,907 rows, 161 columns, and a mixture of numeric and object types, providing a baseline for memory profiling.

Internally, pandas groups columns of the same dtype into blocks; numeric blocks are stored as contiguous numpy.ndarray objects, while object blocks store Python string pointers, leading to higher memory usage.

Numeric columns can be down‑casted by selecting integer columns with DataFrame.select_dtypes(include='int') and applying pd.to_numeric(..., downcast='integer') , reducing integer memory from 7.9 MB to 1.5 MB (≈80% reduction). Float columns are similarly down‑casted from float64 to float32 , halving their memory footprint.

After optimizing numeric types, the overall DataFrame memory drops by about 7 %, prompting a focus on the 78 object columns.

Object columns store Python strings, which are memory‑inefficient because each entry is a pointer to a separate string object. Converting suitable object columns to the category dtype replaces strings with integer codes, dramatically shrinking memory.

For example, the day_of_week column (7 unique values) is converted with df['day_of_week'] = df['day_of_week'].astype('category') . Its underlying representation becomes an int8 array, and memory drops from 9.8 MB to 0.16 MB (≈98% reduction).

A loop iterates over all object columns, checks if the number of unique values is less than 50 % of the column length, and converts qualifying columns to category . In this dataset the conversion reduces total memory from 752.72 MB to 51.667 MB (≈93% reduction), and the full DataFrame shrinks from 861 MB to about 104 MB (≈88% reduction).

Further savings are achieved by specifying optimal dtypes at import time using pandas.read_csv() with a dtype dictionary and parse_dates for date columns, avoiding the need to create a large DataFrame first.

With the optimized DataFrame, simple analyses such as the distribution of game days over decades and the trend of game durations are performed, illustrating both the practical benefits of memory optimization and insights into baseball history.