Python Techniques for Crawling TXT, CSV, PDF, and Word Documents

Introduction

HTML documents are the main type on the web, but other formats such as TXT, WORD, Excel, PDF, and CSV also need to be crawled. This guide records Python‑based methods for fetching these files.

Fetching TXT Files

In Python 3, the common approach is to use urllib.request.urlopen to retrieve the file directly, then apply regular expressions or other techniques to search for sensitive words.

Fetching CSV Files

CSV files can be downloaded similarly; the article includes an illustrative screenshot of the process.

Fetching PDF Files

PDF documents are fetched using the same urllib method, with subsequent processing steps shown in the accompanying image.

Fetching Word Documents

The procedure involves:

Using urlopen to download the remote .docx file.

Converting it to an in‑memory byte stream.

Unzipping the .docx archive (since it is a compressed package).

Reading the extracted files as XML.

Locating the XML tags that contain the main text and processing them.

A disclaimer notes that the content is collected from the internet and the original author retains copyright.