Implementing an SQL Parser: Core Concepts, ANTLR vs. Calcite Comparison, and Practical Code Samples

Background – As big‑data technologies proliferate, many components (e.g., Hive, Spark, Flink) support SQL while others (e.g., Kafka, HBase) do not, creating a need for a unified SQL parsing layer that reduces development effort and maintenance complexity.

Why a SQL parser? – Traditional relational databases cannot handle massive data volumes; a SQL parser abstracts the query interface so a single API can adapt to various data sources, as illustrated by the comparison diagrams in the original article.

What is a SQL parser? – A parser consists of three stages: lexical analysis (tokenizing the input), syntactic analysis (building an abstract syntax tree, AST), and semantic analysis (validating tables, columns, and types). The article shows a simple SQL example:

SELECT name FROM tab;

After lexical analysis the tokens are visualized (image omitted).

Choosing a parser – The two mainstream options are ANTLR and Apache Calcite.

ANTLR – A powerful grammar generator used in many big‑data projects (Hive, Presto, Phoenix). The workflow includes defining .g4 grammar files, generating lexer and parser, traversing the AST, and producing execution plans. Example grammar snippets and Maven dependencies are provided, followed by a Java visitor implementation that evaluates arithmetic expressions:

public class LibExprVisitorImpl extends LibExprBaseVisitor
{
    // implementation omitted for brevity
}

The main class prints the parsed tree:

public class TestLibExprPrint {
    public static void main(String[] args) {
        printTree("E:/smartloli/hadoop/sql-parser-example/src/main/resources/testCase.txt");
    }
    // helper method omitted
}

Calcite – A lightweight framework that focuses on query language, optimization, and execution, delegating storage and data‑management to external engines. Its modules (JDBC client, parser/validator, expression builder, optimizer, etc.) are illustrated with diagrams.

Calcite usage requires only a Maven dependency and a short Java program that defines a schema, loads JSON data, and runs a SQL query:

package com.vivo.learn.sql.calcite;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import org.smartloli.util.JSqlUtils;

public class JSqlClient {
    public static void main(String[] args) {
        JSONObject tabSchema = new JSONObject();
        tabSchema.put("id", "integer");
        tabSchema.put("name", "varchar");
        JSONArray datasets = JSON.parseArray("[{\"id\":1,\"name\":\"aaa\",\"age\":20},...]");
        String sql = "select count(*) as cnt from \"userinfo\"";
        String result = JSqlUtils.query(tabSchema, "userinfo", datasets, sql);
        System.out.println("result: " + result);
    }
}

Comparison – The article presents side‑by‑side tables and screenshots showing that Calcite generally has lower learning cost, simpler integration, and higher flexibility for big‑data use cases, making it the preferred choice for most scenarios.

Conclusion – A SQL parser can unify access to relational databases, provide standard JDBC/ODBC interfaces, empower analysts, and enable SQL on components lacking native support. Selecting the right parser (often Calcite) is crucial for building scalable big‑data query solutions.