Armv8‑A AArch64 Virtualization Overview and Hypervisor Mechanisms

1 Overview

This article describes Armv8‑A AArch64 virtualization support, including stage‑2 page‑table translation, virtual exceptions, and traps. It introduces basic hardware‑assisted virtualization theory and examples of how hypervisors use these features, without detailing any specific hypervisor implementation.

Related Downloads:

ARM CPU Processor Data Collection (1)

ARM CPU Processor Data Collection (2)

ARM Series Processor Application Technology Complete Manual

CPU and GPU Research Framework Collection

1. Industry Deep Report: GPU Research Framework

2. Xinchuang Industry Research Framework

3. ARM Industry Research Framework

4. CPU Research Framework

5. Domestic CPU Research Framework

6. Industry Deep Report: GPU Research Framework

Open‑Source Applications for Arm Architecture Servers

Arm Architecture Servers and Storage

Server Hardware Architecture Overview

Server Market Status Research

1.1 Virtualization Introduction

We introduce basic hypervisor and virtualization theory. A hypervisor is software that creates, manages, and schedules virtual machines (VMs). Readers already familiar with these concepts may skip this section.

Virtualization Why It Matters

Virtualization is widely used in modern cloud computing and enterprise infrastructure. Developers run multiple OS instances on a single hardware platform to develop and test software safely, and most server‑class processors provide virtualization extensions to enable isolation, high availability, load balancing, and sandboxing.

Isolation : Separate VMs on the same physical core cannot access each other's data.

High Availability : Workloads can be migrated transparently between physical machines.

Load Balancing : Distribute workloads across hardware to improve utilization and reduce power consumption.

Sandbox : VMs act as sandboxes to protect host software from buggy or malicious guest applications.

1.2 Two Types of Hypervisor

Hypervisors are classified as Type 1 (bare‑metal) and Type 2 (hosted). Type 2 runs on a host OS that has full control of the hardware; examples include VirtualBox and VMware Workstation. Type 1 runs directly on the hardware without a host OS; examples include Xen and KVM.

Figure 1: Type 2 Hypervisor

The host OS provides the execution environment, while the hypervisor supplies only virtualization functionality.

Conversely, a Type 1 hypervisor runs directly on the hardware and manages resources for guest OSes.

Figure 2: Type 1 Hypervisor

Open‑source hypervisors such as Xen (Type 1) and KVM (Type 2) belong to these two categories.

1.3 Full Virtualization and Paravirtualization

Full virtualization fully emulates hardware, which can be slow for devices like Ethernet. Paravirtualization modifies the guest OS to be aware it runs in a VM, allowing the hypervisor to provide virtual devices for better I/O performance.

Older architectures lacking hardware virtualization required paravirtualization to close security gaps. Modern Arm processors include hardware assistance, so most guests run efficiently without modification, except for some I/O devices that still use virtio or Xen PV Bus.

1.4 Virtual Machine (VM) and Virtual CPU (vCPU)

A VM can be allocated memory pages, which are then visible to all its vCPUs. Virtual interrupts target a specific vCPU, so only that vCPU receives the interrupt.

Figure 3: VM vs vCPU

Note

Arm architecture uses the term Processing Element (PE) for a single execution unit; a vCPU corresponds to a virtual PE.

2 AArch64 Virtualization

In Armv8, the hypervisor runs at Exception Level 2 (EL2). Only software executing at EL2 or higher can access and configure virtualization features.

Stage 2 Translation

EL1/0 Instruction and Register Access

Injecting Virtual Exceptions

Figure 4 shows exception levels and software execution contexts in secure and non‑secure states.

2.1 Stage 2 Translation

What Is Stage 2 Translation?

Stage 2 translation lets the hypervisor control a VM’s memory view, deciding which physical memory a VM can access and where it appears in the VM’s address space. This is essential for isolation and sandboxing.

The hypervisor maintains a Stage 2 page table; the OS maintains a Stage 1 table. The intermediate physical address (IPA) is the address produced by Stage 1 and consumed by Stage 2.

Stage 2 tables have a format similar to Stage 1 but encode memory type attributes directly.

Figure 5: Address Translation VA → IPA → PA

VMID

Each VM receives a VMID stored in VTTBR_EL2 (8 or 16 bits, controlled by VTCR_EL2.vs). VMID tags TLB entries so different VMs can share the same TLB.

VMID vs ASID

ASID tags TLB entries for processes within an OS. A VM may have its own ASID space; VMID and ASID can be used together.

Attribute Merging and Overriding

Stage 1 and Stage 2 mappings each contain attributes (memory type, access permissions). The MMU merges them, choosing the more restrictive attribute.

Figure 6: Attribute Merging Example – Device attribute overrides Normal.

Registers can force attribute overrides, e.g., HCR_EL2.CD forces Stage 1 attributes to Non‑cacheable, HCR_EL2.DC forces Normal Write‑Back, and HCR_EL2.FWB (Armv8.4‑A) lets Stage 2 attributes replace Stage 1.

HCR_EL2.CD – force all Stage 1 attributes to Non‑cacheable.

HCR_EL2.DC – force all Stage 1 attributes to Normal Write‑Back.

HCR_EL2.FWB – use Stage 2 attributes instead of the default restrictive merge.

MMIO Emulation

A VM’s IPA space contains memory and peripheral regions. Virtual devices are fully emulated by the hypervisor.

Figure 7: MMIO Emulation

A pass‑through device is directly mapped into the VM’s IPA space; a virtual device causes a Stage 2 fault that the hypervisor handles.

When handling Stage 1 faults, FAR_ELx reports the faulting virtual address. For Stage 2 faults, HPFAR_EL2 reports the faulting IPA address. ESR_ELx provides exception details (access length, direction, etc.).

Figure 8: Peripheral Device Emulation Flow

VM software accesses a virtual peripheral (e.g., virtual UART FIFO).

Stage 2 translation blocks the access, causing an abort routed to EL2.

Hypervisor reads ESR_EL2 and HPFAR_EL2 to determine the access details.

Hypervisor emulates the device, then returns to the vCPU with ERET.

System Memory Management Units (SMMUs)

DMA controllers also need protection. Without virtualization, DMA sees the same physical address space as the kernel, breaking isolation. By extending Stage 2 translation to DMA, the system uses an SMMU (IOMMU) to enforce the same address translations.

Figure 10: DMA Access Without Virtualization

In a virtualized system, the hypervisor configures the SMMU so that DMA sees the same physical addresses as the kernel, restoring isolation and correct address translation.

Figure 11: DMA Access With Virtualization (No SMMU)

Problems without SMMU: loss of isolation and mismatched address spaces between VM and DMA controller.

Figure 12: DMA Access With SMMU

2.2 Instruction Trapping and Emulation

Hypervisors may need to emulate privileged instructions (e.g., power management, cache control). By configuring traps, such instructions cause an exception that the hypervisor handles.

Example: the WFI (wait‑for‑interrupt) instruction puts the CPU into low‑power mode. If HCR_EL2.TWI is set, executing WFI in EL0/EL1 traps to EL2, where the hypervisor can schedule another vCPU.

Figure 13: WFI Instruction Emulation

2.3 Register Access Trapping

Hypervisors can trap reads of system registers (e.g., ID_AA64MMFR0_EL1) to present virtual values. The hypervisor enables the trap, reads ESR_EL2 for details, writes a virtual value, and returns with ERET.

Figure 14: Register Access Trapping

Frequent register accesses (e.g., MIDR_EL1, MPIDR_EL1) can be expensive if trapped each time. Armv8 provides non‑trapping aliases (VPIDR_EL2, VMPIDR_EL2) that return virtual values without a trap.

VPIDR_EL2 returns a virtual MIDR_EL1 value.

VMPIDR_EL2 returns a virtual MPIDR_EL1 value.

2.4 Virtual Interrupts

Virtual interrupts (vIRQs, vFIQs, vSErrors) are delivered to VMs. Hypervisor configures HCR_EL2 bits (IMO, FMO, etc.) to route physical interrupts to EL2 and then to the appropriate VM.

Enabling Virtual Interrupts

Set HCR_EL2.IMO to enable vIRQ routing, HCR_EL2.FMO for vFIQ, and HCR_EL2.VSE for vSError.

Generating Virtual Interrupts

Two methods: (1) configure HCR_EL2 bits so the CPU generates the interrupt directly; (2) use a GICv2+ interrupt controller to inject virtual interrupts.

Figure 15: GIC Interrupt Injection

Interrupt Forwarding Example

A physical device raises an interrupt, GIC routes it to EL2, hypervisor determines the target VM, configures the GIC to forward it as a virtual interrupt, and the VM receives it.

Figure 16: Virtual Interrupt Forwarding

2.5 Clock Virtualization

Arm provides a generic timer with comparators. EL1 can use a physical timer (wall‑clock) or a virtual timer (virtual counter) whose value is offset per‑vCPU.

Figure 17: Generic Timer Architecture

Figure 18: Two vCPU Scheduling Timeline

EL1 physical timer compares against the system counter (wall‑clock). EL1 virtual timer compares against a virtual counter derived by subtracting a per‑vCPU offset.

Figure 19: Virtual Counter Calculation

Figure 20: Virtual Time Principle

2.6 Virtualization Host Extensions (VHE)

Before VHE, a Type 1 hypervisor needed a high‑visor (EL1) and low‑visor (EL2) split, causing extra context switches. VHE (Armv8.1‑A) allows the host OS kernel to run directly at EL2, eliminating the split.

Figure 21: Type 1 Stack with VHE

Figure 22: Type 2 Stack Before VHE

VHE is enabled by setting HCR_EL2.E2H. The TGE bit selects whether EL0 runs in guest or host address space.

|          Running in          | E2H | TGE |
|-------------------------------|-----|-----|
| Guest kernel (EL1)            |  1  |  0  |
| Guest application (EL0)       |  1  |  0  |
| Host kernel (EL2)             |  1  |  1* |
| Host application (EL0)       |  1  |  1  |
* When a VM exits to the hypervisor, TGE is cleared and must be set before resuming the host kernel.

Figure 23: E2H and TGE Configuration

Virtual Address Space with VHE

Before VHE, EL2 had a single address space. After enabling VHE, EL2 adopts the same split (kernel/user) and ASID support as EL1.

Figure 24: Pre‑VHE EL2 Address Space

Figure 25: EL2 Address Space After Enabling E2H

Redirecting Register Access

When the kernel runs at EL2, accesses to EL1 registers are automatically redirected to EL2. For cases where the hypervisor still needs the original EL1 registers (e.g., task switch), Arm provides alias registers ending with _EL12 or _EL02.

Figure 27: Accessing EL1 Registers from EL2 Using Aliases

2.7 Nested Virtualization

A hypervisor can run inside a VM (guest hypervisor). The host hypervisor runs at EL2, while the guest hypervisor can run at EL1 (Armv8.3‑A) or EL0 (earlier). New control bits HCR_EL2.NV, NV1, NV2 and VNCR_EL2 enable hardware‑assisted nested virtualization and redirect guest hypervisor accesses to a memory region instead of trapping each time.

Figure 28: Nested Virtualization

Guest hypervisor creates a VM, writes configuration to the memory region pointed to by VNCR_EL2, then executes ERET. The host hypervisor reads the configuration, programs the real registers, clears NV bits, and launches the guest VM, reducing the number of traps.

Figure 29: Guest Hypervisor VM Creation (Original Method)

Figure 30: Optimized Guest Hypervisor VM Creation Using VNCR_EL2

2.8 Secure‑World Virtualization

Armv7‑A introduced virtualization only in the non‑secure world. Armv8.4‑A adds EL2 support in the secure world, enabled by SCR_EL3.EEL2. This allows moving firmware from EL3 to EL1 under the protection of virtualization.

Figure 31: Secure‑World Virtualization

Secure EL2 and Two IPA Spaces

Secure world may need two IPA spaces (secure and non‑secure) because Stage 1 can output either secure or non‑secure addresses. Stage 2 does not have an NS bit; a separate register selects which IPA space is used.

Figure 32: Secure‑World IPA Spaces

3 Virtualization Overhead

Switching between VM and hypervisor requires saving/restoring registers: 31 general‑purpose registers, 32 SIMD/FPU registers, and two stack pointers. Using LDP/STP, this costs at least 33 instructions, and overall overhead depends on hardware and hypervisor design.

Source: https://calinyara.github.io/technology/2019/11/03/armv8-virtualization.html