1 views collected around this technical thread.
This article introduces ten practical Python-based web scraping methods—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑plus‑Node reverse engineering—explaining their use cases, advantages, and code examples for efficiently gathering e‑commerce and app data.
This article explains what constitutes high‑quality LLM training data, why large datasets are essential, outlines the step‑by‑step process for collecting, preprocessing, and fine‑tuning models, and highlights the best data sources—including web content, books, code repositories, and news—while noting available free datasets.
This article compiles twenty‑three Python web‑scraping projects, each described with its purpose, key features, and a direct GitHub repository link, offering developers a ready‑made toolbox for data collection across platforms such as WeChat, DouBan, Zhihu, Bilibili, and more.
This article explains how Arkit, a Go‑based unified monitoring agent, collects and parses data from multiple sources, leverages Go plugins for flexible, high‑performance processing, provides custom plugin development guidelines, and discusses the performance benefits and limitations of the plugin system.
This article details Ctrip's approach to weak network detection, covering background, data collection, processing, dynamic weighting algorithms, result output, deployment effects, and future plans, and provides practical code examples and threshold settings for improving mobile network performance.
This article outlines the design, architecture, and practical applications of a quality inspection data collection system, covering data point structures, reporting mechanisms, compliance analysis, intelligent strategy iteration, and BI dashboards, illustrating how big‑data techniques enable digital transformation of inspection processes.
This article presents a detailed case study of Ctrip's large‑scale desktop IT management solution, describing the challenges of handling tens of thousands of office PCs, the full‑link architecture built with Rust, Tauri, SpringBoot and Django, automated health monitoring, fault detection, remediation workflows, security measures, performance optimizations, and the measurable operational improvements achieved.
This article introduces ten practical Python techniques—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑based reverse engineering—to efficiently collect e‑commerce and app data while addressing common challenges such as IP blocking, captchas, and authentication.
This article details the necessity of establishing a no-buried-point scheme mindset, key advantages over traditional methods, mobile client no-buried-point collection implementation mechanisms, data processing, and the benefits of this approach.
This article outlines practical steps—understanding problem background, gathering quality data, selecting appropriate models, solving and analyzing them, and applying results—to ensure mathematical modeling moves beyond theory and effectively addresses real-world issues.
The visual event tracking solution replaces costly manual code instrumentation with a Data‑Trackid and relative Xpath system, a VSCode plugin for automatic ID generation, and an SDK that captures clicks and exposures, dynamically loads third‑party analytics, and provides validation, monitoring, and future decoupling for scalable, real‑time product analytics.
This article explains what frontend event tracking (埋点) is, why it is essential for product analytics, when and how to implement it, the different tracking models and reporting methods, as well as practical tips, iteration processes, and common pitfalls for developers and product teams.
This article provides a comprehensive guide to front‑end event tracking, covering its definition, motivations, scenarios, various tracking types, data models, reporting mechanisms, implementation steps, security considerations, and practical tips for ensuring accurate and non‑blocking data collection in web applications.
This article explains how to collect images from Zhihu's "beauty" topic using Python's Requests and lxml libraries, filter them with Baidu AI's AipFace face detection service, and store the qualified pictures locally, detailing the required environment, logic, and preparation steps.
This article explains what event tracking (埋点) is, its role in collecting user behavior data, the real‑time and offline data pipelines, the responsible teams, the end‑to‑end implementation process, common quality issues, detailed test cases, and the Track platform used for verification.
This article presents a comprehensive overview of Zhihu's event‑tracking system, covering its motivation, toolset, demand‑management platform, verification workflow, data‑collection pipeline, query service architecture, cloud‑native data service design, and practical Q&A on best practices and optimization strategies.
The article explains how cloud‑native observability systems gather massive telemetry from infrastructure, containers, middleware and services, compares direct push and file‑based collection approaches, and details head, tail and local sampling methods to optimize data completeness and performance.
The article explains that while observability is not new, cloud‑native systems have driven rapid development of observable platforms, detailing data collection architectures, direct push versus file‑based approaches, and various sampling techniques (head, tail, and local sampling) to balance completeness, real‑time reporting, and performance impact.
Vivo’s Points Task System evolved from a simple configuration‑driven task model into a scalable, multi‑source behavior incentive platform that uses an AviatorScript engine, unified SDK, and three isolated services—event collection, computation, and task handling—to deliver configurable tasks, real‑time rewards, and flexible user notifications while ensuring stability and extensibility.
This article presents a comprehensive overview of EMQ's Neuron industrial IoT data collection platform, detailing the lessons learned from version 1.x, the redesigned v2.0 architecture, core modules, plugin mechanisms, data‑tag management, eKuiper integration, and two real‑world case studies in oil‑field and smart‑factory environments.