1 views collected around this technical thread.
The article details how to add field‑lineage tracking to Spark SQL by creating a custom SparkSessionExtension that injects a check‑analysis rule and a parser, which capture INSERT statements, analyze the physical plan, and generate a JSON mapping of source‑to‑target fields for data governance.
The Baidu Search Closed‑Door Technical Symposium, the first core technical forum hosted by Baidu’s Search Architecture Department, brings senior engineers and junior backend developers together to discuss semantic retrieval, data‑driven big‑data processing, and vertical search offline architecture, while offering limited‑capacity sessions, networking gifts, and travel subsidies.
This article details the background, architectural evolution, design challenges, and distributed implementation of an accounting system that automates the processing of millions of transaction records across thousands of accounts, highlighting how splitting accounts, workflows, and bills improves performance and reliability.