Data Lake Challenges and the Open SPL Computing Engine

Data warehouses integrate multi‑system data for predefined analytical queries, but they struggle with unforeseen questions because new queries require costly re‑ingestion and transformation of raw data.

Data lakes emerged to store massive raw data of all structures, preserving original information and theoretically enabling any future analysis; however, processing especially structured data still relies heavily on SQL‑based database technologies and ETL pipelines, leading to the so‑called “lake‑warehouse” integration.

The core dilemma—called the data‑lake impossible triangle—is that a lake must simultaneously keep raw data, support convenient computation, and remain inexpensive, yet current implementations can satisfy at most two of these goals.

The open‑source SPL (Structured Processing Language) engine is presented as a solution, offering an open, multi‑source computation layer that can directly operate on raw data in the lake, whether stored in native formats or files, without requiring full ETL into a warehouse.

SPL supports heterogeneous sources (RDBMS, NoSQL, JSON/XML, CSV, Web services) for mixed‑source calculations, provides robust file‑based computation, and includes a rich library of functions that match SQL’s expressiveness while simplifying complex operations.

For high‑performance storage, SPL offers two formats: collection files with compression and segment‑wise parallelism, and group tables with columnar storage and min‑max indexes, enabling efficient parallel execution and fast Top‑N queries.

By leveraging SPL, organizations can bypass traditional data‑warehouse bottlenecks, perform parallel and mixed computations on both curated and raw data, and achieve scalable, cost‑effective data‑lake analytics.

Additionally, the article announces the formation of a free, ad‑free SPL community group for interested technologists.

Example SPL code snippets:

=json(file("/data/EO.json").read())

=A1.conj(Orders)

=A2.select(Amount>1000 && Amount<=3000 && like@c(Client,"*s*"))

=A2.groups(year(OrderDate);sum(Amount))

=A1.new(Name,Gender,Dept,Orders.OrderID,Orders.Client,Orders.Client,Orders.SellerId,Orders.Amount,Orders.OrderDate)