Answers: How Does a Hypervisor Work?¶
1. Which part runs a VM's processors, which plays its devices, and which starts and stops it?¶
KVM runs the VM's processors. It's the part in the Linux kernel: it switches the real processor into guest mode to run one of the VM's virtual processors, and takes control back whenever the guest reaches for the real machine. With the processor's help, it also keeps each VM's memory separate.
Which virtual processor gets a real one at any moment? By default, the Linux kernel decides, just as it does for your programs.
QEMU plays the disk and the network card. Each running VM is one QEMU process, which presents the VM with devices its drivers recognise and does the real work with the host's own hardware.
libvirt starts and stops the VM. Its service, libvirtd, launches a VM's QEMU process, stops it, and keeps track of which VMs are running.
Why three parts? Each one does a different job. NIST splits any hypervisor platform the same way: a core for the processors and memory, pieces for the devices, and management pieces that start and stop VMs.
2. What happens when Ubuntu, inside a VM, sends data to its disk?¶
Step by step:
- Ubuntu's kernel reaches for its disk. A disk is a device, something only the hypervisor may touch, so the processor stops the guest for a moment and hands control to KVM.
- KVM passes the request on to the VM's QEMU process.
- QEMU writes the data to the VM's disk on the host: a file, a device or storage reached over the network.
- QEMU reports the write done to Ubuntu.
Why can't Ubuntu write to the real disk itself? Because it never sees it. It runs in guest mode, where the hypervisor stays in charge, and the disk its drivers talk to is a virtual one that QEMU plays.
That's the whole point of the arrangement. If every guest could command the real disk, three kernels would be giving orders to the same hardware, and nothing would keep the VMs apart.
3. What does hardware virtualisation spare the hypervisor, and what does it cost? Why does CloudStack refuse a KVM host without it?¶
It spares the rewriting. Without the processor's help, one way out is for the hypervisor to find the guests' instructions that touch the machine itself, and translate them in software. With it, the guest's own code runs straight on the processor, in guest mode, and the processor hands control to the hypervisor when it matters. Less code runs with full privileges, and an unchanged operating system runs with its usual drivers.
The cost is the hand-over itself: every switch to the hypervisor takes time, and NIST calls these switches the main cause of slowdown in a virtualised system. Only some operations cause one.
CloudStack refuses such a host because KVM can't run without it on Intel and AMD processors, and every VM CloudStack starts on a KVM host asks for KVM. CloudStack never falls back to running VMs with QEMU alone. So when a KVM host is added, CloudStack's setup on that machine checks that KVM can run there, and stops if it can't; on Ubuntu, the check is kvm-ok (see the Evidence below).
4. Whose process is the one ps shows per VM, and what does restarting libvirtd do to the VMs?¶
It's QEMU's. Each running VM is one QEMU process, which holds the VM's virtual devices and its memory, while KVM, in the kernel, runs its processors. That's exactly what CloudStack's own script looks for when it searches the process list for qemu.
If an admin restarts libvirtd, the VMs keep running. The manual says so: "Restarting libvirtd does not impact running guests."
That tells you where a VM really lives: in its QEMU process, with KVM's help. libvirt manages VMs; it doesn't run them.
5. Type 1 or type 2: ESXi, VirtualBox, Xen? And why is KVM hard to label?¶
ESXi is type 1: it's installed straight on the bare machine, with no ordinary operating system underneath.
Xen is type 1 too, in NIST's list (see the Evidence below). It's the hypervisor inside XenServer and XCP-ng.
VirtualBox is type 2: it's an application running on an ordinary operating system, like any other program on a laptop.
KVM is hard to label because it has something of both. It's part of the Linux kernel, which, with KVM loaded, becomes the hypervisor's core, right on the hardware, like type 1. But the machine is still an ordinary Linux system, where other programs can run beside the VMs and compete with them for the hardware, like type 2.
NIST says KVM has "features of both"; ShapeBlue calls it type 1. The safest answer describes it instead: a full Linux kernel turned hypervisor, with ordinary programs alongside.
And neither label is CloudStack's "hypervisor type", which is simply which hypervisor a host runs.
Evidence¶
The post states these facts in plain words. Here are the lines of CloudStack's code, and the passages of the docs, that back them. Code is quoted at the commit the post is written against.
A hypervisor platform is a core, device pieces and management pieces¶
NIST SP 800-125A Rev. 1, Executive Summary (final text):
The core software used for server virtualization is called the Hypervisor which directly provides Central Processing Unit (CPU) and memory virtualization. […] Since the hypervisor by itself cannot provide all functions needed for server virtualization, it has supporting software modules (e.g., device drivers) for devices (e.g., Network and Storage devices) virtualization in addition to management modules for VM lifecycle operations and hypervisor configuration. The hypervisor together with these supporting modules and the hosting hardware constitute the hypervisor platform.
The same document, section 1, compares the hypervisor with an operating system: "Just as a conventional OS provides isolation between the various applications (or processes) running on a server, the hypervisor provides isolation between one or more VMs running on it."
Guest mode, the hand-over, and what the processor's help spares¶
NIST SP 800-125A Rev. 1, section 4.1, on the two modes:
Hardware extensions for Virtualization provide two modes of execution: host or root mode and guest or non-root mode. The host mode runs at a higher privilege than guest mode. The hypervisor code, which provides the baseline functionality HY-BF1 (processor allocation and memory management), runs in host mode while the guest OS and applications in VMs run in guest mode.
The post calls host mode "the processor's more powerful mode", and avoids the word "root", which means something else on Linux.
The same section, on what the help spares, and on memory:
Additionally, when the hypervisor does not have to perform additional functions (e.g., translating sensitive instructions using techniques such as binary translation), the code executing with privileges is reduced in the hypervisor, making the TCB smaller and enabling better assurance verification.
The consequence of this feature is that an unmodified Guest OS, along with their native device drivers, can be run in VMs.
Memory Virtualization: Hardware-assisted memory virtualization is provided when the hardware enables the mapping of the Guest OS's physical addresses in their respective page tables to the host's physical addresses using hardware-based page tables instead of hypervisor-generated shadow page tables.
NISTIR 8221, section 3 (final text), names the hand-over a "VM exit" and says only some operations cause one: "VM exits are a response to some instructions and events (e.g., page fault) from guest VMs and are the main cause of performance degradation in a virtualized system. These events could include external interrupts, accesses to control registers, task switches, and I/O operation instructions".
CloudStack checks each KVM host for KVM before using it¶
CloudStack's KVM docs require "HVM (Intel-VT or AMD-V enabled)" on every KVM host (Host KVM Installation).
When a KVM host is added, the management server logs in to it and runs CloudStack's setup script there (LibvirtServerDiscoverer.java, lines 300–304):
// cloudstack: server/src/main/java/com/cloud/hypervisor/kvm/discoverer/LibvirtServerDiscoverer.java, lines 300–304
String setupAgentCommand = "cloudstack-setup-agent ";
if (!username.equals("root")) {
setupAgentCommand = "sudo cloudstack-setup-agent ";
}
if (!SSHCmdHelper.sshExecuteCmd(sshConnection, setupAgentCommand + parameters)) {
The setup checks for KVM first, and stops if the check fails (syscfg.py, lines 137–139):
# cloudstack: python/lib/cloudutils/syscfg.py, lines 137–139
kvmEnabled = self.svo.isKVMEnabled()
if not kvmEnabled:
raise CloudInternalException("Checking KVM...[Failed]\nPlease enable KVM on this machine\n")
On Ubuntu, the check is kvm-ok, the command from the first article's lab section (utilities.py, lines 234–235). On other distributions it looks for the device file /dev/kvm instead.
# cloudstack: python/lib/cloudutils/utilities.py, lines 234–235
def isKVMEnabled(self):
return bash("kvm-ok").isSuccess()
Each running VM is one QEMU process¶
The post shows how CloudStack's heartbeat script finds VMs among the host's qemu processes. To delete those VMs, the script's next lines kill each process it found, one by one (kvmheartbeat.sh, lines 89–92):
# cloudstack: scripts/vm/hypervisor/kvm/kvmheartbeat.sh, lines 89–92
for pid in $vmPids
do
kill -9 $pid &> /dev/null
done
NISTIR 8221, section 2.1.2, says that on KVM "VMs run as normal Linux processes", which is why the kernel shares the processors among them like any other program. It also describes KVM this way: "the KVM module uses QEMU to create guest VMs running as separate user processes". Its Figure 2 draws one QEMU inside each VM. (The order in that sentence reads backwards: it's QEMU that asks KVM to run the VM, as the next section shows.)
The VM's memory belongs to that process too. The Linux kernel's own KVM documentation says a guest's memory is taken from an address that "must point at user addressable memory", the ordinary memory of a program. And the requests about a VM "must be issued from the same process (address space) that was used to create the VM": QEMU's.
QEMU plays the devices, and asks KVM to run the processors¶
QEMU's manual on Ubuntu 24.04 (qemu-system-x86_64) opens with "The QEMU PC System emulator simulates the following peripherals", a list that includes "2 PCI IDE interfaces with hard disk and CD-ROM support" and "PCI and ISA network adapters".
Ubuntu's docs (VM tools in the Ubuntu space) put the two parts together: "QEMU primarily functions as the user-space backend for KVM. When used in collaboration with KVM kernel components, it harnesses the hardware virtualization capability that KVM provides in order to efficiently virtualize guests."
QEMU is the one that asks. Its manual's -accel option: "This is used to enable an accelerator. Depending on the target architecture, kvm, xen, hvf, nvmm, whpx or tcg can be available. By default, tcg is used." (TCG is QEMU's own software emulator.) The kernel's KVM documentation describes the request QEMU makes for each virtual processor: "This ioctl is used to run a guest virtual cpu."
Without an accelerator, QEMU imitates the processor too. QEMU's own docs: "Note that the default is TCG, which is purely emulated, so you must specify an accelerator type to take advantage of hardware virtualization."
CloudStack never runs VMs with QEMU alone: every VM asks for KVM¶
When CloudStack's program on a KVM host builds a VM's description, it picks the VM's type. The first branch is only for hosts that run LXC containers instead of VMs (LibvirtComputingResource.java, lines 3541–3547):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java, lines 3541–3547
protected void configureGuestAndVMHypervisorType(VirtualMachineTO vmTO, LibvirtVMDef vm, GuestDef guest) {
if (HypervisorType.LXC == hypervisorType && VirtualMachine.Type.User.equals(vmTO.getType())) {
configureGuestAndUserVMToUseLXC(vm, guest);
} else {
configureGuestAndSystemVMToUseKVM(vm, guest);
}
}
On a KVM host, every VM takes the second branch, which sets the type to kvm (LibvirtComputingResource.java, lines 3552–3554):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java, lines 3552–3554
private void configureGuestAndSystemVMToUseKVM(LibvirtVMDef vm, GuestDef guest) {
guest.setGuestType(GuestDef.GuestType.KVM);
vm.setHvsType(HypervisorType.KVM.toString().toLowerCase());
That type opens the description CloudStack hands to libvirt (LibvirtVMDef.java, line 2685). libvirt's own word for a VM is "domain" (nothing to do with CloudStack's own domains, which come later in this module):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java, line 2685
vmBuilder.append("<domain type='" + _hvsType + "'>\n");
libvirt runs each VM with QEMU¶
CloudStack's program connects to libvirt's QEMU driver (LibvirtConnection.java, line 115):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java, line 115
return "qemu:///system";
libvirt's QEMU driver page says what that driver runs: "It supports multiple QEMU accelerators: software emulation also known as TCG, hardware-assisted virtualization on Linux with KVM…".
On Ubuntu, the libvirt page of Ubuntu's docs lists a running guest as the process "/usr/bin/qemu-system-x86_64".
libvirt starts, stops and watches VMs; restarting it doesn't stop them¶
The libvirtd manual on Ubuntu 24.04 (libvirtd):
This daemon runs on host servers and performs required management tasks for virtualized guests. This includes activities such as starting, stopping and migrating guests between host servers, configuring and manipulating networking, and managing storage for use by guests.
The libvirt client libraries and utilities connect to this daemon to issue tasks and collect information about the configuration and resources of the host system and guests.
Restarting libvirtd does not impact running guests.
Ubuntu's QEMU page gives the reason libvirt exists: "Although QEMU has a command line interface and a monitor to interact with running guests, they are typically only used for development purposes. On the other hand, libvirt provides an abstraction from specific versions and hypervisors, and encapsulates some workarounds and best practices."
CloudStack uses libvirt through its own program, and sends the full description at each start¶
CloudStack's program on a KVM host runs as a service that needs libvirtd (cloudstack-agent.service, lines 22–26):
# cloudstack: packaging/systemd/cloudstack-agent.service, lines 22–26
[Unit]
Description=CloudStack Agent
Documentation=http://www.cloudstack.org/
Requires=libvirtd.service
After=libvirtd.service
It starts each VM from a full description, which it doesn't let libvirt keep (LibvirtComputingResource.java, lines 2345–2354):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java, lines 2345–2354
public String startVM(final Connect conn, final String vmName, final String domainXML, int flags) throws LibvirtException, InternalErrorException {
try {
/*
We create a transient domain here. When this method gets
called we receive a full XML specification of the guest,
so no need to define it persistent.
This also makes sure we never have any old "garbage" defined
in libvirt which might haunt us.
*/
"Transient" means libvirt holds the description only while the VM runs. A few lines on, the method hands the description to libvirt, which starts the VM (LibvirtComputingResource.java, line 2373):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java, line 2373
conn.domainCreateXML(domainXML, flags);
libvirt is a separate project¶
CloudStack's KVM code uses libvirt's own Java library, published by the libvirt project (plugins/hypervisors/kvm/pom.xml, lines 45–49):
<!-- cloudstack: plugins/hypervisors/kvm/pom.xml, lines 45–49 -->
<dependency>
<groupId>org.libvirt</groupId>
<artifactId>libvirt</artifactId>
<version>${cs.libvirt-java.version}</version>
</dependency>
CloudStack's build fetches it from libvirt.org (pom.xml, lines 847–848):
<!-- cloudstack: pom.xml, lines 847–848 -->
<id>libvirt.org.maven2</id>
<url>https://libvirt.org/maven2/</url>
On a CloudStack host, QEMU does the work for every disk¶
Each disk in a VM's description names QEMU as its driver (LibvirtVMDef.java, lines 1366–1375):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java, lines 1366–1375
public String toString() {
StringBuilder diskBuilder = new StringBuilder();
diskBuilder.append("<disk ");
if (_deviceType != null) {
diskBuilder.append(" device='" + _deviceType + "'");
}
diskBuilder.append(" type='" + _diskType + "'");
diskBuilder.append(">\n");
if(qemuDriver) {
diskBuilder.append("<driver name='qemu'" + " type='" + _diskFmtType + "' ");
qemuDriver is on by default (LibvirtVMDef.java, line 977):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtVMDef.java, line 977
private boolean qemuDriver = true;
At this commit, the code switches it off in one place only (LibvirtComputingResource.java, line 3955):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java, line 3955
diskdef.setQemuDriver(false);
That line sits inside the branch for LXC containers, not VMs, which the post leaves out (LibvirtComputingResource.java, lines 3936–3937):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java, lines 3936–3937
// For LXC, find and add the root filesystem, rbd data disks
if (HypervisorType.LXC.toString().toLowerCase().equals(vm.getHvsType())) {
NIST describes the general path for an emulated device (SP 800-125A Rev. 1, section 2.2.2): "An I/O call from a guest VM application (through its guest OS) is intercepted by the hypervisor kernel and forwarded to this code since guest VMs cannot access the physical devices directly under this setup."
Xen is the hypervisor inside XenServer and XCP-ng¶
CloudStack's code for the XenServer family reads the version of Xen, the hypervisor, from each host (XcpServerDiscoverer.java, lines 300–302):
// cloudstack: plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/discoverer/XcpServerDiscoverer.java, lines 300–302
String prodVersion = CitrixHelper.getProductVersion(record);
String xenVersion = record.softwareVersion.get("xen");
String hostOS = record.softwareVersion.get("product_brand");
The same code serves hosts whose product is XenServer, XCP-ng or Citrix Hypervisor (XcpServerDiscoverer.java, line 443):
// cloudstack: plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/discoverer/XcpServerDiscoverer.java, line 443
} else if (prodBrand.equals("XenServer") || prodBrand.equals("XCP-ng") || prodBrand.equals("Citrix Hypervisor")) {
Type 1, type 2, and KVM's place¶
NIST's ITL Bulletin for March 2018 (PDF): "The hypervisor can be installed either directly on the hardware, or bare metal (Type 1 Hypervisor), or on top of a full-fledged conventional OS, called Host OS (Type 2 Hypervisor)."
NISTIR 8221, section 2.1:
Type 1 hypervisors run directly on the host's hardware to control the hardware and manage guest operating systems (Guest OS). For this reason, Type 1 hypervisors are sometimes called bare metal hypervisors and include Xen, Microsoft Hyper-V, and VMware ESX/ESXi. Type 2 hypervisors are similar to other computer programs that run on an OS as a process. […] Some systems have features of both. For example, Linux's Kernel-based Virtual Machine (KVM) is a kernel module that effectively converts the host OS to a Type 1 hypervisor but is also categorized as a Type 2 hypervisor because Linux distributions are still general-purpose OSs with other applications competing for VM resources.
ShapeBlue (Choosing the Right Hypervisor): "Kernel-based Virtual Machine (KVM) is a Type 1 hypervisor integrated into the Linux kernel".
CloudStack's "hypervisor type" is which hypervisor a host runs¶
In CloudStack's code, a hypervisor type is a name such as XenServer, KVM or VMware, each with its own disk-image format (Hypervisor.java, lines 47–49):
// cloudstack: api/src/main/java/com/cloud/hypervisor/Hypervisor.java, lines 47–49
public static final HypervisorType XenServer = new HypervisorType("XenServer", ImageFormat.VHD, EnumSet.of(RootDiskSizeOverride, VmStorageMigration));
public static final HypervisorType KVM = new HypervisorType("KVM", ImageFormat.QCOW2, EnumSet.of(DirectDownloadTemplate, RootDiskSizeOverride, VmStorageMigration));
public static final HypervisorType VMware = new HypervisorType("VMware", ImageFormat.OVA, EnumSet.of(RootDiskSizeOverride, VmStorageMigration, VmStorageMigrationWithSnapshots));
Nothing in it says type 1 or type 2.