Answers: What Does a Cloud Add to Virtual Machines?¶
1. Five KVM hosts and VMs on request by email: is that a cloud?¶
No. It's virtualisation, done by hand.
KVM and libvirt do run VMs, but each host's libvirt manages that host's VMs and nothing else. It starts a VM where it's asked; it never chooses a host. Someone still has to read each email, pick a host, log in and ask.
Two things are missing, and they're the two traits of NIST's definition the article followed:
- Self-service. A person sits between every request and every VM, so nothing happens at 2 a.m., and the queue grows with every new researcher.
- A shared pool, handed out automatically. The hosts are there, but nothing places each VM where there's room, keeps track of whose VM is where, or stops one group taking everything.
What would add them is software above the hosts: something that takes each request itself, picks a host, keeps the records and enforces the rules. NIST's own footnote says a cloud is hardware plus that software. In CloudStack, it's the management server.
2. Self-service and a shared pool take the admin out of every request. What must the software do instead?¶
Everything the admin used to do, with no person in the loop, at any hour:
- Hold each tenant to its share: check, for every new VM, that its tenant hasn't reached its limit.
- Pick a host with room: one with enough spare CPU and memory for the size asked for. The person asking chooses only a place; only the operators may name a host.
- Keep the records: whose each VM is, and which host it's on.
- Tell the chosen host to start the VM. On a KVM host, the instruction ends at that host's libvirt, the same libvirt the admin used by hand.
- Refuse requests that touch another tenant's VMs.
That list is the management server's job description. Steps 2 to 4 even come in that order in the code: it picks the host, writes it into the VM's record, and only then sends the instruction (see the Evidence below).
3. The cloud's only management server crashes at 3 a.m. What happens to the running VMs, and what can't anyone do?¶
The running VMs carry on. CloudStack's docs say so even for the worst case, every copy of the management server down: "All Guest Instances will continue to work."
Why? Because the management server isn't in the VMs' path. Each VM runs under its own host's hypervisor, which doesn't need the management server to keep it going. On a KVM host, CloudStack's own program there notices the connection has dropped, and simply keeps trying to reconnect. A VM's own work, such as serving a web page, never went through the management server in the first place.
What stops is every request to the cloud. Nobody can create a VM, and the web interface doesn't answer. And if a host fails meanwhile, the automatic restarts that CloudStack can do for that host's VMs don't happen while it's down either.
In the kitchen picture: HQ has closed for the night. The kitchens keep cooking the brands already open, but no new brand can launch until morning.
4. What is a tenant, and can a physicist stop the Biology VM on the same host?¶
A tenant is one of the customers that share a cloud, kept apart from the others: usually a group, such as a company or a department of a large organisation. It isn't the same thing as a person: it owns the VMs, and its people each log in as members of it. And it needn't pay: at the university, Physics and Biology are tenants, and they're departments.
The physicist can't stop the Biology VM. Sharing a host is normal (it's what makes the pool a pool), but sharing a host isn't sharing VMs. Two guards stand between the tenants' VMs here (their networks are kept apart too):
- On the host, the hypervisor keeps the two VMs apart, as it keeps every VM apart from its neighbours: strongly, though not absolutely.
- At the management server, a request that names a VM is checked against that VM's owner. Physics doesn't own the Biology VM, so the request is refused.
Why must tenants be kept apart at all? Because the pool is shared. Without the checks, any tenant could touch, or take, what's another's, and no tenant could trust the cloud. Tenants who need more than that can have hosts reserved for them alone, but that's an option the operators switch on, not the rule.
5. Is CloudStack a hypervisor?¶
No. CloudStack commands hypervisors; the hypervisor on each host runs the VMs.
Its management server decides where each VM goes and sends the instruction. What happens next depends on the family:
- On a KVM host, a small CloudStack program on the host hands the instruction to the host's libvirt, which starts the VM.
- For the XenServer family (XenServer and XCP-ng), CloudStack talks to XAPI, the family's own management service.
- For VMware, CloudStack talks to vCenter, which manages the ESXi hosts.
Even in the smallest set-up, where one machine runs the management server and is also a KVM host, the management server stays a program of its own, separate from the hypervisor. And the VMs outlive the management server's outages: if it were their hypervisor, stopping it would stop them.
Evidence¶
The post states these facts in plain words. Here are the lines of CloudStack's code, and the passages of the docs, that back them. Code is quoted at the commit the post is written against.
Each host's libvirt manages that host's VMs¶
CloudStack's program on a KVM host connects to libvirt with a local address (LibvirtConnection.java, line 115):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java, line 115
return "qemu:///system";
libvirt's Connection URIs page explains it: "qemu:///system connects to a system mode daemon", which "will handle qemu instances on behalf of all users of the machine (among other things)". An address with no host name in it is local: "Either the transport or the hostname must be given in order to distinguish this from a local URI." One libvirt, one machine.
NIST's two traits, and what a cloud physically is¶
NIST SP 800-145 (final text), section 2, on self-service:
On-demand self-service. A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service provider.
On the shared pool:
Resource pooling. The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
Its footnote 2, on hardware plus software:
A cloud infrastructure is the collection of hardware and software that enables the five essential characteristics of cloud computing. The cloud infrastructure can be viewed as containing both a physical layer and an abstraction layer. The physical layer consists of the hardware resources that are necessary to support the cloud services being provided, and typically includes server, storage and network components. The abstraction layer consists of the software deployed across the physical layer, which manifests the essential cloud characteristics. Conceptually the abstraction layer sits above the physical layer.
And its example of the many consumers inside one organisation, from the same section: "The cloud infrastructure is provisioned for exclusive use by a single organization comprising multiple consumers (e.g., business units)."
CloudStack manages VMs, and companies run it for themselves¶
CloudStack's README (README.md, lines 57–62):
# cloudstack: README.md, lines 57–62
Apache CloudStack is open source software designed to deploy and manage large
networks of virtual machines, as a highly available, highly scalable
Infrastructure as a Service (IaaS) cloud computing platform. CloudStack is used
by a number of service providers to offer public cloud services, and by many
companies to provide an on-premises (private) cloud offering, or as part of a
hybrid cloud solution.
It's licensed under the Apache License 2.0 (LICENSE, lines 4–5):
# cloudstack: LICENSE, lines 4–5
Apache License
Version 2.0, January 2004
And the About page says: "Apache CloudStack is a top-level project of the Apache Software Foundation (ASF)."
A tenant is a customer or a department; sharing hosts is the default¶
CloudStack's docs (Accounts) describe CloudStack's record for a tenant, which it calls an account: "An Account typically represents a customer of the service provider or a department in a large organization."
Hosts for one tenant alone are an option (Dedicating Resources to Accounts and Domains): "The root administrator can dedicate resources to a specific domain or Account that needs private infrastructure for additional security or performance guarantees." The root administrators are the cloud's operators. One of the kinds of dedication reads "A host will not be shared across multiple Accounts", which only needs saying because, without it, hosts are shared.
The management server is one service, on a machine of its own or a VM¶
The service's name, in its unit file (cloudstack-management.service, line 26):
# cloudstack: packaging/systemd/cloudstack-management.service, line 26
Description=CloudStack Management Server
Where it runs (Management Server Overview): "The management server typically runs on a dedicated machine or as a virtual machine." And the smallest set-up (Deployment Architecture Overview): "In its smallest deployment, a single machine can act as both the Management Server and the hypervisor host (using the KVM hypervisor)."
Several copies (High Availability): "The Management Server itself may be deployed in a multi-node installation where the servers are load balanced."
Ordinary users may create and delete VMs by themselves¶
A fresh install's default rules give the role called "User" (role 4) these two commands. (The row's description, "Default Root Admin role", is a slip in the code, not its meaning.) The role (schema-481to490.sql, line 473):
-- cloudstack: engine/schema/src/main/resources/META-INF/db/schema-481to490.sql, line 473
INSERT INTO `cloud`.`roles` (`id`, `uuid`, `name`, `role_type`, `description`) values (4, UUID(), 'User', 'User', 'Default Root Admin role') ON DUPLICATE KEY UPDATE name=name;
Creating a VM (create-default-role-api-mappings.sql, line 734):
-- cloudstack: engine/schema/src/main/resources/META-INF/db/create-default-role-api-mappings.sql, line 734
INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 4, 'deployVirtualMachine', 'ALLOW', 93) ON DUPLICATE KEY UPDATE rule=rule;
And deleting one (create-default-role-api-mappings.sql, line 736):
-- cloudstack: engine/schema/src/main/resources/META-INF/db/create-default-role-api-mappings.sql, line 736
INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) values (UUID(), 4, 'destroyVirtualMachine', 'ALLOW', 95) ON DUPLICATE KEY UPDATE rule=rule;
A request names a size, a place and what to start from¶
A size is required (DeployVMCmd.java, lines 55–57):
// cloudstack: api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java, lines 55–57
@ACL
@Parameter(name = ApiConstants.SERVICE_OFFERING_ID, type = CommandType.UUID, entityType = ServiceOfferingResponse.class, required = true, description = "The ID of the Service offering for the Instance")
private Long serviceOfferingId;
So is a place (BaseDeployVMCmd.java, line 79):
// cloudstack: api/src/main/java/org/apache/cloudstack/api/command/user/vm/BaseDeployVMCmd.java, line 79
@Parameter(name = ApiConstants.ZONE_ID, type = CommandType.UUID, entityType = ZoneResponse.class, required = true, description = "availability zone for the virtual machine")
And, unless the VM starts blank, exactly one starting point, such as an operating system image (DeployVMCmd.java, lines 241–244):
// cloudstack: api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java, lines 241–244
public void create() throws ResourceAllocationException {
if (!isBlankInstance() && Stream.of(templateId, snapshotId, volumeId).filter(Objects::nonNull).count() != 1) {
throw new CloudRuntimeException("Please provide only one of the following parameters - template ID, volume ID or snapshot ID");
}
The words in these lines (offering, zone, template) are taught later in the module.
Each tenant's limit: 20 VMs by default¶
The default (Config.java, lines 1171–1178):
// cloudstack: server/src/main/java/com/cloud/configuration/Config.java, lines 1171–1178
DefaultMaxAccountUserVms(
"Account Defaults",
ManagementServer.class,
Long.class,
"max.account.user.vms",
"20",
"The default maximum number of user VMs that can be deployed for an account",
null),
Creating a VM reserves one against the owner's limit (UserVmManagerImpl.java, line 4483):
// cloudstack: server/src/main/java/com/cloud/vm/UserVmManagerImpl.java, line 4483
try (CheckedReservation vmReservation = new CheckedReservation(owner, ResourceType.user_vm, resourceLimitHostTags, 1l, reservationDao, resourceLimitService);
The operators' own accounts aren't limited, which is why the limit is a tenant's, not the operators' (ResourceLimitManagerImpl.java, lines 654–657):
// cloudstack: server/src/main/java/com/cloud/resourcelimit/ResourceLimitManagerImpl.java, lines 654–657
// Don't place any limits on system or root admin accounts
if (_accountMgr.isRootAdmin(account.getId()) && !(considerSystemAccount && Account.ACCOUNT_ID_SYSTEM == account.getId())) {
return;
}
The management server picks a host with enough CPU and memory¶
When a VM starts, the management server plans where it goes (VirtualMachineManagerImpl.java, lines 1473–1476):
// cloudstack: engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java, lines 1473–1476
DeployDestination dest = null;
try {
dest = _dpMgr.planDeployment(vmProfile, plan, avoids, planner);
} catch (final AffinityConflictException e2) {
A host is kept only if it has the CPU and memory the size asks for (CapacityManagerImpl.java, lines 1069–1077):
// cloudstack: server/src/main/java/com/cloud/capacity/CapacityManagerImpl.java, lines 1069–1077
int cpu_requested = offering.getCpu() * offering.getSpeed();
long ram_requested = offering.getRamSize() * 1024L * 1024L;
Pair<String, String> clusterDetails = getClusterValues(host.getClusterId());
float cpuOvercommitRatio = Float.parseFloat(clusterDetails.first());
float memoryOvercommitRatio = Float.parseFloat(clusterDetails.second());
boolean hostHasCpuCapability = checkIfHostHasCpuCapability(host, offering.getCpu(), offering.getSpeed());
boolean hostHasCapacity = checkIfHostHasCapacity(host, cpu_requested, ram_requested, false, cpuOvercommitRatio, memoryOvercommitRatio,
considerReservedCapacity);
Only the operators may name a host¶
If anyone but a root admin, one of the cloud's operators, names a host when creating or starting a VM, the request is refused (UserVmManagerImpl.java, lines 6267–6273):
// cloudstack: server/src/main/java/com/cloud/vm/UserVmManagerImpl.java, lines 6267–6273
private HostVO getDestinationHost(Long hostId, boolean isRootAdmin, boolean isExplicitHost, boolean quickRestore) {
HostVO destinationHost = null;
if (hostId != null) {
if (isExplicitHost && !isRootAdmin && !quickRestore) {
throw new PermissionDeniedException(
"Parameter " + ApiConstants.HOST_ID + " can only be specified by a Root Admin, permission denied");
}
The path that creates and starts a VM passes true for isExplicitHost (UserVmManagerImpl.java, lines 5996–6000); quickRestore is a special case, for restoring from a backup:
// cloudstack: server/src/main/java/com/cloud/vm/UserVmManagerImpl.java, lines 5996–6000
@Override
public Pair<UserVmVO, Map<VirtualMachineProfile.Param, Object>> startVirtualMachine(long vmId, Long podId, Long clusterId, Long hostId,
@NotNull Map<VirtualMachineProfile.Param, Object> additionalParams, String deploymentPlannerToUse, boolean quickRestore)
throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException, ResourceAllocationException {
return startVirtualMachine(vmId, podId, clusterId, hostId, additionalParams, deploymentPlannerToUse, true, quickRestore);
Decide, record, instruct, in that order¶
Each VM's record names its host (VMInstanceVO.java, lines 107–108):
// cloudstack: engine/schema/src/main/java/com/cloud/vm/VMInstanceVO.java, lines 107–108
@Column(name = "host_id", updatable = true, nullable = true)
protected Long hostId;
And its owner (VMInstanceVO.java, lines 151–152):
// cloudstack: engine/schema/src/main/java/com/cloud/vm/VMInstanceVO.java, lines 151–152
@Column(name = "account_id")
protected long accountId;
Once the host is chosen (the planning above, line 1475), the VM's change of state writes the chosen host into its record (VirtualMachineManagerImpl.java, line 1510):
// cloudstack: engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java, line 1510
if (!changeState(vm, Event.OperationRetry, destHostId, work, Step.Prepare)) {
Only then, further down the same method, is the start instruction sent to that host (VirtualMachineManagerImpl.java, line 1578):
// cloudstack: engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java, line 1578
_agentMgr.send(destHostId, cmds);
On a KVM host, the instruction ends at libvirt¶
On a KVM host, this class of CloudStack's program handles the start instruction (StartCommand) (LibvirtStartCommandWrapper.java, lines 55–56):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java, lines 55–56
@ResourceWrapper(handles = StartCommand.class)
public final class LibvirtStartCommandWrapper extends CommandWrapper<StartCommand, Answer, LibvirtComputingResource> {
It starts the VM through libvirt (LibvirtStartCommandWrapper.java, line 97):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java, line 97
libvirtComputingResource.startVM(conn, vmName, vmFinalSpecification);
That method hands the VM's description to libvirt, as the hypervisor article's answers show.
A request on another tenant's VM is refused¶
For a request that names a VM, the management server checks the caller against the VM's owner. Root admins pass, and so does the owner (DomainChecker.java, lines 234–243):
// cloudstack: server/src/main/java/com/cloud/acl/DomainChecker.java, lines 234–243
protected void validateCallerHasAccessToEntityOwner(Account caller, ControlledEntity entity, AccessType accessType) {
PermissionDeniedException exception = new PermissionDeniedException("Caller does not have permission to operate with provided resource.");
if (_accountService.isRootAdmin(caller.getId())) {
return;
}
if (caller.getId() == entity.getAccountId()) {
return;
}
After two more rules, for administrators of part of the cloud and for shared projects (both taught later in the course), everyone else is refused (DomainChecker.java, lines 275–276):
// cloudstack: server/src/main/java/com/cloud/acl/DomainChecker.java, lines 275–276
logger.error("Caller: {} does not have permission to access {}", caller, entityLog);
throw exception;
XAPI and vCenter: the other families' own management software¶
For the XenServer family, CloudStack's code calls XAPI (CitrixResourceBase.java, lines 185–186):
// cloudstack: plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java, lines 185–186
* CitrixResourceBase encapsulates the calls to the XenServer Xapi process to
* perform the required functionalities for CloudStack.
For VMware, CloudStack's docs (Host VMware vSphere Installation) say: "CloudStack requires ESXi and vCenter." vCenter must be "available to manage the vSphere hosts", and reachable "so that it can communicate with the CloudStack Management Server."
The VMs keep running when the management server stops¶
CloudStack's Concepts page (Massively Scalable Infrastructure Management): "Maintenance or other outages of the management server can occur without affecting the Instances running in the cloud."
What stops, from the reliability page the post quotes (HA for Management Server): "When the Management Server is down, no new Instances can be created, and the end User and admin UI, API, dynamic load distribution, and HA will cease to work." The UI is the web interface; HA is the automatic restarts after a host fails. (The same section also calls the management server itself "stateless". That's too strong: each running copy also holds live connections to some of the hosts, as How Do You Ask CloudStack for Something? (coming soon) shows. Only its lasting records are kept elsewhere.)
On a KVM host, the code agrees. When CloudStack's program on the host loses its connection, it runs its disconnect steps, logs the loss and tries to reconnect (Agent.java, lines 640–642):
// cloudstack: agent/src/main/java/com/cloud/agent/Agent.java, lines 640–642
serverResource.disconnected();
logger.info("Lost connection to host: {}. Attempting reconnection while we still have {} commands in progress.", shell.getConnectedHost(), commandsInProgress.get());
stopAndCleanupConnection(true);
It keeps redialling until a management server answers (Agent.java, lines 648–651):
// cloudstack: agent/src/main/java/com/cloud/agent/Agent.java, lines 648–651
do {
if (CollectionUtils.isEmpty(avoidMSHostList) || !avoidMSHostList.contains(host)) {
connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(), shell.getSslHandshakeTimeout(), this);
logger.info("Reconnecting to host: {}", host);
On the way, the only thing it does is cancel a move that was in progress: a VM's to another host, or a disk's to other storage. Those are the only two disconnect steps in the code (LibvirtMigrateCommandWrapper.java, lines 247–249):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateCommandWrapper.java, lines 247–249
// add cancel hook before we start. If migration fails to start and hook is called, it's non-fatal
cancelHook = new MigrationCancelHook(dm);
libvirtComputingResource.addDisconnectHook(cancelHook);
and (LibvirtMigrateVolumeCommandWrapper.java, lines 144–145):
// cloudstack: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateVolumeCommandWrapper.java, lines 144–145
cancelHook = new VolumeMigrationCancelHook(dm, destDiskLabel);
libvirtComputingResource.addDisconnectHook(cancelHook);
Nothing on that path stops a running VM.