Detailed Overview of LiteIO Architecture, Components, and Volume Lifecycle

The previous two articles introduced LiteIO's features and deployment methods; this piece focuses on the specific functions and collaboration principles of each LiteIO component and explains the lifecycle of a volume within the system.

LiteIO's components are divided into core functions (node‑disk‑controller, disk‑agent, nvmf_tgt) and CSI functions (csi‑controller, csi‑node). Core functions handle resource scheduling, lifecycle management, and operations such as volume creation, scheduling, expansion, deletion, and StoragePool reporting. CSI functions expose these capabilities to Kubernetes via the Container Storage Interface, turning LiteIO volumes into PersistentVolumes (PV) and PersistentVolumeClaims (PVC).

From a user perspective, creating a Pod with a LiteIO PVC triggers a series of events: the pod is scheduled, the PVC is bound, the csi‑controller creates a LiteIO volume, node‑disk‑controller schedules the volume to a node, disk‑agent provisions the volume on the node, and kubelet mounts it for the pod. Deleting the pod and PVC reverses these steps, ultimately freeing the resources.

PART 02 – Disk‑Agent The Disk‑Agent runs on each storage node and is responsible for discovering or building StoragePools, reporting their status, maintaining heartbeats, managing volumes, snapshots, migration tasks, and exposing IO metrics. It supports two pool engines: LVM (kernel‑based) and SPDK‑LVS (requiring an nvmf_tgt process).

Error 1 – Early implementations called the Agent directly via RPC from node‑disk‑controller, leading to duplicated logic, tight coupling, and network isolation issues. The current design lets the Agent interact only with the Kubernetes API server, using CRD watches to decouple components.

Error 2 – The heartbeat implementation failed to reset hs.lease after an update error, causing agents to stop sending heartbeats. The fix is to set hs.lease = nil or re‑fetch the Lease after a failed update.

Agent configuration determines how StoragePools are built (e.g., file‑based LVM, SPDK aio bdev, or VFIO‑based NVMe). Example YAML configuration for an aio‑LVS pool is shown below:

storage:
  pooling:
    name: aio-lvs
    mode: SpdkLVStore
  bdev:
    type: aioBdev
    name: test-aio-bdev
    size: 1048576000 # 1GiB
    filePath: /local-storage/aio-lvs

PART 03 – Node‑Disk‑Controller This centrally deployed component maintains the cluster-wide state of resources, schedules volumes, and coordinates migration and snapshot tasks. It uses controller‑runtime reconcilers and a plug‑in architecture (PlugableReconciler) to keep synchronization logic modular.

Metadata synchronization to an external database is performed directly within the node‑disk‑controller's reconcilers, avoiding a separate sync controller and reducing resource overhead.

The built‑in scheduler loads all resources into memory via an EventHandler, then filters nodes using basic and affinity filters and ranks them with priority functions, including considerations for PositionAdvice and local‑disk preferences.

PART 04 – CSI Components The csi‑controller implements volume create/delete/expand and snapshot APIs, working with the csi‑provisioner sidecar. The csi‑node daemonset runs on each compute node, handling StageVolume (connect remote disk, format, mount) and PublishVolume (bind‑mount to the container). Deletion follows the reverse UnpublishVolume and UnstageVolume steps. Metrics for volume capacity and inode usage are exposed via Prometheus, and a CLI tool is provided for manual CSI operations.