1 views collected around this technical thread.
This tutorial walks you through building a Python web‑scraper for Beike (Lianjia) second‑hand property listings, covering session spoofing, dynamic parameters, pagination, multithreaded detail fetching, data cleaning, and exporting results to Excel in a step‑by‑step manner.
Learn how to efficiently scrape web pages using Python by exploring multiple approaches—including simple requests with BeautifulSoup, fast parsing with lxml, dynamic content extraction with Selenium, and large‑scale crawling with Scrapy—complete with installation steps, code snippets, and detailed explanations.
This tutorial explains how to use Python to scrape product reviews from JD.com via its AJAX comment API, extract fields such as nickname, score, content, and image count, and save the collected data into a CSV file using the requests and csv modules.
This article introduces ten practical Python-based web scraping methods—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑plus‑Node reverse engineering—explaining their use cases, advantages, and code examples for efficiently gathering e‑commerce and app data.
This article explains why proxy IPs are essential for reliable web crawling, compares dynamic and static residential proxies, and provides step‑by‑step Python code to scrape product titles, prices and links from Snapdeal while demonstrating how to integrate proxies for improved efficiency and security.
This article outlines Python's advantages for web crawling, introduces core libraries such as Requests, BeautifulSoup, and Scrapy, details a step-by-step development workflow, provides practical code examples for extracting news titles, and highlights important considerations and advanced techniques for robust scraper implementation.
This article introduces beginner-friendly Python web‑scraping methods, covering the simplest urllib/requests approach, adding request headers, inspecting network traffic for hidden data, handling dynamically loaded content with Selenium, and provides links to deeper tutorials for each technique.
Learn how Python can interact with the three major API types—SOAP, REST, and GraphQL—by understanding their core concepts, differences, and practical code examples using libraries like zeep and requests, empowering you to build robust integrations across diverse services.
This article introduces the fundamentals of the HTTP protocol, explains its request and response structures, and provides a comprehensive guide to using Python's requests library for sending GET, POST, PUT, DELETE requests, handling headers, parameters, JSON responses, timeouts, and exceptions.
This tutorial walks through building a Python script that fetches a QQ Music QR code, displays it locally, monitors its status, extracts required encrypted parameters from cookies, and completes the login process by repeatedly polling the authentication endpoint to obtain a session.
This guide walks through setting up a Python environment, organizing a test project, writing reusable request utilities and data‑driven Pytest cases, generating Allure reports, and extending tests with token handling and database verification for robust backend API automation.
This article explains how to use Python's requests, BeautifulSoup, and Pillow libraries to crawl high‑resolution anime pictures from the Huashi6 website, extract their webp URLs via regex, download them, and convert the images to the more common png format for personal use.
This tutorial walks through multiple Python techniques for downloading regular files, web pages, Amazon S3 objects, and other resources, covering basic requests, wget, handling redirects, chunked large‑file downloads, parallel downloads, progress bars, urllib, urllib3, proxy usage, boto3 for S3, and asynchronous downloads with asyncio.
This article demonstrates how to use Python's requests library to perform chained API calls—login, create user, retrieve user details, and update user information—while handling exceptions, logging, and environment configuration for robust automated testing.
This guide explains why request headers are needed, shows how to set them using Python's requests library—including basic headers and authentication tokens—and demonstrates how to view both request and response headers along with common header usage tips.
This tutorial explains the Python requests library, covering its core modules, HTTP methods (GET, POST, PUT, DELETE), key parameters, session management, and exception handling with practical code examples for API interaction and best practices.
This article introduces web crawling fundamentals, demonstrates how to inspect HTML elements, walks through simple examples using urllib, requests, and BeautifulSoup, and culminates in a complete Python script that extracts chapter links and contents from a novel website, saving them to a text file.
This article provides a collection of practical Python snippets demonstrating how to send GET and POST requests, validate status codes, response bodies, JSON schemas, handle headers, query parameters, file uploads, redirects, timeouts, sessions, authentication, rate limiting, and parameterized testing using the requests library and pytest framework.
This article introduces the open‑source HttpRunner API testing tool, outlines its key features such as YAML/JSON test definitions, parameterization, assertions, reporting and performance testing, and provides step‑by‑step Python examples showing how to write, run, and integrate HttpRunner tests with existing requests‑based projects.
This tutorial explains how to capture Bilibili video streams with Fiddler, extract the necessary request headers and URLs, and use a concise Python script based on the requests library to download videos, handling header nuances and batch‑fetching video lists from an uploader’s page.