Trends and Challenges in Data‑Intensive High‑Performance Computing (HPC) and AI Workloads

Over the past 25 years the domestic general‑purpose HPC market has expanded from $7.2 billion in 1996 to about $27.9 billion in 2019 and is projected to reach $37.7 billion in 2024, making it one of the fastest‑growing IT markets worldwide. Growth is fueled by increasing computational demand from scientific and engineering research, national competition for the fastest supercomputers, and commercial technologies that democratize HPC.

Recent market acceleration is especially evident in data‑intensive HPC (HPDA) and AI workloads. While the overall HPC market is expected to grow at a 6.8 % CAGR (2019‑2024), the HPDA segment is forecast to expand at an average 17 % CAGR, and the AI sub‑segment at an even higher 33 % CAGR.

The shift from traditional compute‑intensive modeling and simulation to data‑intensive AI/ML/DL workloads has turned storage into a critical component, now accounting for roughly 20 % of total HPC spend and projected to generate $8 billion in revenue for on‑premise HPC storage by 2024.

Hyperion Research notes that the proliferation of data‑intensive applications is reshaping the HPC ecosystem: HPDA/AI workloads generate massive data volumes that stress existing storage solutions, requiring new architectures that can handle both large sequential and small random I/O patterns.

Global “E‑class” supercomputer competition is driving multi‑hundred‑million‑dollar investments in ultra‑high‑performance systems, with governments viewing HPC as a strategic resource and emphasizing domestic supply chains.

HPC workloads are increasingly moving to the cloud. Most users view cloud computing as a complement rather than a replacement for on‑premise HPC, adopting hybrid environments that use containers to orchestrate compute, network, and storage across private HPC clouds and public cloud services. By 2024, cloud‑based HPC spending is expected to reach $8.8 billion, of which $2.9 billion will be spent on cloud storage.

The article defines key terms: AI (broadly encompassing machine learning, deep learning, and other methods), ML (training models on labeled examples), and DL (advanced neural‑network‑based learning without explicit programming).

Challenges for data‑intensive HPC storage include the need to support mixed I/O models (large sequential and small random), multi‑protocol access (S3, NFS, MPI‑IO, SMB, HDFS), tiered data handling for hot versus cold datasets, and high‑density, cost‑effective designs that address power, cooling, and rack space constraints.

Opportunities lie in unified storage architectures that deliver both high bandwidth (TB/s) for large sequential accesses and high IOPS for random workloads, support for multiple access protocols within a single system, and flexible tiering using SSDs and HDDs to match data access frequency, thereby reducing total cost of ownership for the HPC community.