Analyzing 1.4 Billion N‑gram Rows with Python, NumPy and PyTubes

The Google Ngram Viewer visualizes word usage over time using a huge corpus of scanned books; the author reproduces the Python‑keyword trend by downloading the public n‑gram dataset (covering the 16th‑century to 2008) and processing it with Python.

Because the 1‑gram files total 27 GB on disk (about 1.43 billion rows across 38 files), the author leverages NumPy’s fast array operations and a new data‑loading library called PyTubes to read the tab‑separated data on a modest 8 GB, 2016 MacBook Pro.

After extracting the three fields (word, year, count) and filtering for the capitalised form “Python”, the script builds a NumPy array of the relevant rows, computes yearly total word counts, and derives the percentage of “Python” occurrences per year.

Using these percentages, the author recreates the Google Ngram chart for Python, then extends the analysis to compare three programming languages—Python, Pascal, and Perl—by normalising counts to a common baseline (1800‑1960 average) and plotting their relative trends.

The author notes performance differences (Google’s chart renders in ~1 s versus the script’s ~8 min) and suggests future improvements to PyTubes, such as supporting smaller integer dtypes, richer filtering combinators, and enhanced string‑matching utilities.