Understanding Spark SQL Parsing Layer and Its Optimizations

The presentation is the third session of a Spark series. The first two sessions covered Spark Core fundamentals and Spark SQL architecture; this session focuses on the parsing layer, which is the first stage of Spark SQL processing and is relatively simple to understand.

Product Introduction : Two main products are introduced – CyberEngine (a cloud‑native data‑lake foundation that manages Spark SQL) and CyberData (a unified data‑development platform supporting batch‑stream, lake‑warehouse, and multi‑cloud environments).

Spark SQL Parsing Layer Principles : The execution flow of Spark SQL passes through the parsing layer, optimization layer, and execution plan layer. The parser is built on ANTLR4, which generates lexer and parser components from .g4 grammar files (SqlBaseLexer.g4 and SqlBaseParser.g4). The parsing layer produces an abstract syntax tree (AST) that later stages consume.

The ANTLR4 grammar defines tokens (SELECT, FROM, etc.) and syntax rules (singleStatement, query, queryOrganization). After compilation, ANTLR4 generates interfaces and abstract classes such as SqlBaseParserBaseVisitor and SqlBaseParserVisitor . Spark implements these via DataTypeAstBuilder and AstBuilder , which build logical plans (LogicPlan) from the AST.

Core Implementation Classes : The parsing entry point is SqlBaseParser , whose parent AbstractParser defines the parse method. The visitQuery method in AstBuilder handles SELECT queries, processing clauses such as ORDER BY, WINDOW, LIMIT, and CTE. The resulting LogicPlan is a tree representation of the query.

Optimization Case : The speaker contributed two syntax PRs to Spark 3.2, adding percentile_cont and percentile_disc . By refactoring the generic functionCall logic, the implementation was reduced from ~30 lines to a few lines, improving code elegance and maintainability.

The session concluded with a summary of the parsing layer, a preview of upcoming optimization topics, and a Q&A covering metrics, SQL optimization evaluation, vectorization vs. native execution, ANTLR4 learning resources, and Spark’s future direction.