Build Systems and Bundlers: Concepts, Rebuilders, Schedulers, and Incremental Compilation

While researching Rspack's incremental implementation, I discovered the paper Build Systems à la Carte: Theory and Practice and used it as a basis to explain build systems and relate them to modern bundlers.

Build System

A build system automates a series of repeatable tasks. Classic examples include Make , Shake and Bazel , which take source files and a task‑description file (e.g., a Makefile) to produce executables. Less common examples are Excel , which treats cells as inputs and formulas as tasks, and UI frameworks that treat props as inputs and components as tasks.

From these examples we can extract a set of generic concepts:

Task – the unit of work, described by a task descriptor (e.g., a Makefile or an Excel formula).

Input – data consumed by a task.

Output – data produced by a task, possibly feeding other tasks.

Info – build‑time information that can be reused in the next build (e.g., file modification timestamps, which act like a cache).

Store – the place where inputs, outputs and info are stored (the file system for Make, a database for other systems).

Build – the process of taking defined tasks and an existing store, feeding new inputs, and producing a new store.

The main differences between build systems stem from two decisions:

Whether a task should be re‑executed.

In which order tasks should be executed.

These decisions correspond to the concepts of Rebuilder and Scheduler . Different build systems are essentially different combinations of rebuilder and scheduler.

Scheduler

A scheduler decides the order of task execution. Common strategies are:

Topological – execute tasks according to a topological sort of static dependencies.

Restarting – pick a task, and if its dependencies are not yet satisfied, pick another task until all are done.

Suspending – when a task's dependencies are missing, first execute those dependencies, then resume the original task (easily expressed with async/await).

Rebuilder

A rebuilder decides whether a task needs to run again. Typical approaches are:

Dirty‑bit – each task records a clean/dirty flag; after a build all are clean, and changed inputs mark the corresponding tasks dirty.

Verifying traces – store lightweight metadata (hashes or timestamps) and re‑run a task only if its dependencies changed.

Constructive traces – like verifying traces but also store the actual content, enabling cloud caching and remote execution.

Build Systems as Rebuilder + Scheduler Combinations

Examples:

make = topological modTimeRebuilder – static dependencies, topological scheduler, dirty‑bit rebuilder based on file modification times.

excel = restarting dirtyBitRebuilder – dynamic dependencies, restarting scheduler, dirty‑bit rebuilder.

bazel = restarting ctRebuilder – restarting scheduler with cloud‑trace rebuilder for remote execution.

shake = suspending vtRebuilder – suspending scheduler, verifying‑trace rebuilder that enables minimality and early cutoff.

cloudShake = suspending ctRebuilder – same as Shake but with constructive‑trace rebuilder for cloud caching.

buck2 = suspending ctRebuilder – supports dynamic dependencies, minimality, early cutoff, cloud cache and remote execution.

Bundlers

A bundler can be seen as a build system plus a partial task descriptor. Early task‑runners such as gulp and grunt were essentially build systems. Modern bundlers (Webpack, Parcel, Rollup, esbuild) describe part of the task logic (module graph construction, chunking, optimisations) while the user configuration and plugins supply the rest.

Key differences between bundlers and classic build systems:

Bundlers have highly dynamic dependencies – a module’s code may depend on the result of another module’s generation.

Bundlers must handle cycles in the module graph, which many traditional build systems cannot.

Bundlers also distinguish two kinds of builds: watch‑mode incremental rebuilds (memory cache) and full rebuilds (persistent cache).

Pass‑Based Bundlers (Webpack/Parcel/Rollup/esbuild)

passBasedBundler = foreach ctRebuilder

Each pass has its own scheduler and rebuilder. For example, Webpack uses a suspending scheduler for side‑effects optimisation and a restarting scheduler with a verifying‑trace rebuilder for export handling. Most passes use a simple “foreach” scheduler combined with a cloud‑trace rebuilder, providing minimality but lacking early cutoff.

Turbopack

turbopack = suspending ctRebuilder

Turbopack adopts a query‑based model: tasks are defined and fetched on demand, avoiding the need for a full pass pipeline in development. Its underlying incremental engine (Turbo Tasks) implements the full build‑system stack, and Turbopack adds a cloud‑trace rebuilder to achieve both minimality and early cutoff.

Vite

vite = suspending vtRebuilder

Vite does not bundle; it compiles individual modules on demand using the browser’s native ESM loader. It follows a suspending scheduler with a verifying‑trace rebuilder, but is limited by browser constraints (concurrency limits, non‑shareable cache).

Rspack

incrementalRspack = foreach dirtyBitAndCtRebuilder

Rspack is a pass‑based bundler that introduces “affected‑based incremental” computation: each stage records the changes that affect later stages, allowing only the impacted tasks to be re‑executed. This adds an early‑cutoff rebuilder to the existing foreach scheduler, moving Rspack closer to self‑adjusting computation.

Summary

Many bundlers claim to be “next‑generation”, yet from the build‑system perspective they often lack long‑standing features such as minimality, early cutoff, parallelism, remote cache and remote execution. Incorporating these proven concepts can substantially improve bundler performance and scalability.

References

Build Systems à la Carte: Theory and Practice – https://www.microsoft.com/en-us/research/uploads/prod/2020/04/build-systems-jfp.pdf

Make – https://www.gnu.org/software/make/

Shake – https://shakebuild.com/

Bazel – https://bazel.build/

Excel – https://en.wikipedia.org/wiki/Microsoft_Excel

Build‑system combinations – https://github.com/snowleopard/build/blob/43b18b9a362d7d27b64679ea4122e4b8c5dfedd9/src/Build/System.hs

Shake author – https://ndmitchell.com/

Buck2 vs Cloud Shake – https://www.reddit.com/r/rust/comments/136qs44/comment/jipq5pj/

Remote execution – https://buck2.build/docs/users/remote_execution/

DICE – https://github.com/facebook/buck2/blob/3fd1dbec7a212291c222b803a43a1355b48c3fb9/dice/dice/docs/index.md

gulp – https://gulpjs.com/

grunt – https://gruntjs.com/

Buck2 DICE integration – https://ndmitchell.com/downloads/paper-implementing_applicative_build_systems_monadically-01_jan_2022.pdf

Affected‑based incremental – https://github.com/web-infra-dev/rspack/discussions/8243

Self‑Adjusting Computation – https://arc.net/l/quote/yjoxhuft