Understanding MPI, OpenMPI, OpenMP and the Differences Between SMP, NUMA, and MPP Architectures

HPC systems are essentially parallel‑computing platforms, and beginners often confuse MPI, OpenMPI, and OpenMP because the terms sound similar and their definitions overlap.

MPI (Message Passing Interface) is a language‑independent communication protocol (standard) implemented as a library. Implementations include MPICH, MPI‑1, MPI‑2, OpenMPI, IntelMPI, PlatformMPI, etc. OpenMPI (Open Message Passing Interface) is one specific implementation of MPI.

OpenMP (Open Multiprocessing) is an application programming interface that provides a shared‑memory parallel programming model. In modern parallel systems both OpenMP and OpenMPI are needed: OpenMP handles intra‑node (shared‑memory) parallelism, while OpenMPI manages inter‑node (distributed‑memory) communication.

The commercial servers in use today can be roughly classified into three architectures: SMP (Symmetric Multi‑Processor) , NUMA (Non‑Uniform Memory Access) , and MPP (Massive Parallel Processing) .

These architectures feature two main memory models: UMA (Uniform Memory Access) and NUMA (Non‑Uniform Memory Access) . Variants such as COMA and ccNUMA are improvements on the basic NUMA design.

SMP (Symmetric Multi‑Processor)

SMP systems contain tightly coupled processors that share all resources (bus, memory, I/O). The operating system sees a single instance, and all CPUs access memory and peripherals equally. Resource contention is resolved by hardware/software locking mechanisms.

Because all resources are shared, SMP servers have limited scalability; memory bandwidth becomes a bottleneck as CPU count grows. Empirical tests show optimal CPU utilization with 2‑4 CPUs.

NUMA (Non‑Uniform Memory Access)

NUMA allows dozens or even hundreds of CPUs to be combined in a single server, overcoming SMP’s scalability limits. Each CPU module has its own local memory and I/O slots, while inter‑module communication occurs via a cross‑bar switch.

Local memory accesses are much faster than remote ones, so applications should minimize cross‑module data exchange. NUMA can support hundreds of CPUs in a single physical server, but performance does not scale linearly because remote memory latency is high.

MPP (Massive Parallel Processing)

MPP consists of many SMP nodes connected by a high‑speed interconnect. Each node accesses only its own local resources (memory, storage) and runs its own OS and database instance. This “share‑nothing” design provides excellent scalability, theoretically unlimited, with current technology supporting up to 512 nodes and thousands of CPUs.

In MPP systems, data redistribution between nodes replaces remote memory accesses. Products such as Teradata hide the complexity of node scheduling and load balancing behind a unified relational database interface.

Each processing unit in an MPP system has private CPU, bus, memory, and storage, and runs its own OS and database replica, ensuring no resource sharing across nodes.

Performance Differences Between NUMA, MPP, and SMP

NUMA’s inter‑node communication occurs within a single physical server; remote memory accesses incur latency, preventing linear performance scaling as CPUs increase.

MPP’s nodes are separate SMP servers linked via I/O; each node accesses only local memory, allowing near‑linear performance growth when adding nodes.

SMP shares all CPU resources, so true linear scaling is not achievable due to shared memory bandwidth constraints.

Application Differences Between MPP, SMP, and NUMA

NUMA excels in OLTP workloads where many CPUs share a single server, but remote memory latency makes it less suitable for data‑warehouse workloads that require heavy inter‑CPU data exchange.

MPP’s lack of shared resources makes it more efficient for large‑scale decision‑support and data‑mining tasks, provided inter‑node communication is minimal.

When communication overhead is high, SMP can outperform MPP because all CPUs share the same memory space.

For further reading, see the e‑book “Evolution of High‑Performance Computing (HPC) Technologies, Ecosystem, and Industry Trends” and additional resources on HPC fundamentals, applications, and data‑replication technologies.

>>>>>>>>>>>>> Reader Benefits <<<<<<<<<<<<<<

Knowledge Extension Reading

HPC Basics: Fundamental Knowledge

HPC Main Applications and Software

Data Replication Management (CDM) Technology Analysis and Product Recommendations

Warm Reminder: Search for “ICT_Architect” or scan the QR code to follow the public account and click the original link for more HPC technical materials.