Web Scraping E‑commerce Site with Selenium and Python: From Page Analysis to MySQL Storage

The article begins by showing how to open the target e‑commerce site, search for the keyword "men's clothing", and inspect the network requests to locate the GET URL containing the query parameters.

It then explains how to use Selenium to request the page, wait for the search input to appear, enter the keyword, and click the search button, ensuring each element is clickable before proceeding.

Next, the tutorial covers pagination: retrieving the total number of result pages, navigating to a specific page by entering the page number, clicking the pagination button, and verifying that the active page is highlighted.

After reaching each result page, the script waits for the product list to load, extracts the page source, and parses it with PyQuery. For each product item, it collects the shop name, location, title, price, sales volume, and image URL, assembling them into a list.

The collected data is then saved into a MySQL database. The code creates a table (named after the search keyword), constructs an INSERT statement, and inserts each record, handling commit/rollback and closing the connection.

Finally, the article provides the complete Python script that ties together the functions get_one_page(name) , get_next_page(page, name) , get_info(name) , and save_to_mysql(data, name) , and runs the whole process for the keyword "男装" (men's clothing).

All code snippets are presented in full, wrapped in ... tags, and the article ends with a link to the original blog post for further reading.