0 views collected around this technical thread.
This article introduces NetEase’s Arctic, a real‑time lakehouse system built on Apache Iceberg that unifies streaming and batch processing, explains the challenges of Lambda architecture, details Arctic’s features such as change/base stores, hidden queue, transaction handling, and shares internal practice cases and future roadmap.
This article presents NetEase's big data platform architecture, detailing multi‑layer storage and compute design, HDFS deployment challenges, NameNode and NameSpace performance optimizations, cluster scaling strategies, data tiering, hardware upgrades, and real‑world business use cases, illustrating practical large‑scale big data engineering.
This article introduces NetEase's internally incubated data lake project Arctic, explains the concept of data lakes, outlines NetEase's specific requirements for a unified streaming‑batch platform, details Arctic's core architecture, storage strategy, data‑merge mechanisms, current achievements, and future development plans.
The presentation at the DataFun Summit detailed NetEase's data‑governance tool practice, reviewing past initiatives, current challenges, comprehensive product strategies, and future roadmap to improve compute and storage efficiency, cost quantification, and systematic governance across business lines.
This article describes how NetEase Cloud Music leveraged Serverless function computing to redesign its audio/video algorithm processing platform, covering the existing challenges, the selection criteria for Serverless solutions, the implementation details, performance gains, cost savings, and future directions.
This article presents an in‑depth overview of NetEase's massive big‑data ecosystem, its diverse product lines, the full‑link data platform matrix, user‑profile taxonomy, ID‑mapping techniques, regional domain insights, tag management, quality assurance mechanisms, and real‑world case studies demonstrating applications in marketing, recommendation, growth, and anti‑fraud scenarios.
This article explains Apache Kudu’s architecture, schema design, update mechanism, and how NetEase leverages it for real‑time data ingestion, dimension table joins, data‑warehouse ETL, and AB‑testing, while also discussing encountered issues and upcoming feature requests.
This article explains how to use Python's requests and lxml libraries to scrape NetEase Cloud Music's song list, extract song IDs, construct download URLs, and save the audio files locally, while also discussing common obstacles such as anti‑scraping measures and network errors.
The Youzan Big Data Technology Salon brought together Youzan, NetEase and Didi to share practical approaches for cutting data‑infrastructure costs, building an Apache Iceberg‑based data lake, scaling Flink real‑time workloads, and creating a data‑driven growth platform that leverages tracking, A/B testing and analytics.
This article details NetEase's evolution from Storm to Flink for real‑time computing, describing the Sloth platform's architecture, large‑scale deployment, diverse business scenarios, monitoring, alerting, and future development plans, illustrating how Flink powers data synchronization, real‑time warehousing, and e‑commerce analytics and recommendation.
In March 2020, Apple devices displayed security warnings because the HTTPS certificate for appleimap.163.com had expired due to NetEase Mail’s failure to renew it, highlighting how overlooked certificate management can disrupt services and underscoring the need for automated monitoring tools in large organizations.
This article, originally shared by NetEase’s data expert Jiang Hongxiang on DataFun, outlines the construction of NetEase’s data foundation platform, covering database kernel insights and the implementation of the ad‑hoc query engine Impala with the distributed storage system Kudu, offering valuable big‑data engineering practices.
This article presents NetEase Hangzhou Research Institute's experience in building a data infrastructure, covering database innovations such as InnoSQL, NTSDB, and InnoRocks, as well as the integration of big‑data components like HDFS, Spark, Impala, and Kudu to enable efficient storage, processing, and real‑time analytics.