Skip to content
CodeFlowerHorn
Menu

CloudStack Setup · Guide 06

How to Set Up High Availability and Failover in Apache CloudStack

Configure KVM Host HA, IPMI-based out-of-band management, and HA-enabled instances so CloudStack can recover more safely from compute-host failures.

CodeFlowerHornApache CloudStack 4.19
Table of contents+

Introduction

Understand CloudStack high availability before enabling it

Apache CloudStack has two related but different HA concepts. HA-enabled instances are monitored so CloudStack can restart them after a confirmed failure, while Host HA monitors KVM compute hosts and uses out-of-band power management to recover or fence a failed host.

Instance HA

Restart the workload

When CloudStack determines that an HA-enabled instance has failed, it can attempt to restart that instance on another suitable host in the same cluster.

Host HA

Recover or fence the host

The KVM Host HA framework performs health and activity checks, attempts recovery through OOBM, and can fence a host when recovery fails.

Before you begin

  • • Use at least two KVM hosts in the same CloudStack cluster.
  • • Both hosts should use the same network layout and be able to access the same shared primary storage.
  • • This guide continues with the shared NFS primary storage configured earlier in the series.
  • • Each physical host needs working out-of-band management such as IPMI, iDRAC, or iLO.
  • • The Management Server must be able to reach the BMC/IPMI management address of every HA-enabled host.
  • • Test power-management commands before relying on Host HA for real workloads.
!

HA is not a replacement for backup or application-level redundancy

Host HA and instance restart automation can reduce recovery time after infrastructure failures, but they do not protect you from filesystem corruption, accidental deletion, application errors, or complete storage failure.

01

IPMI settings

Configure the server's out-of-band management interface

Start on the physical server's management controller. On Dell hardware this is normally iDRAC; other vendors may provide iLO, IPMI, or another BMC interface. Enable IPMI-over-LAN or the equivalent remote power-management feature and assign the management controller a stable IP address.

SettingExampleNotes
BMC / iDRAC IP192.168.10.11Use a dedicated management address, not the KVM host OS address.
ProtocolIPMI 2.0CloudStack's IPMITOOL driver normally uses the lanplus interface.
Port623 / UDPDefault IPMI RMCP port.
AccountDedicated CloudStack BMC userUse only the permissions needed for power-state operations.

If you are using Dell iDRAC, the exact menu varies by generation, but the setting is typically under iDRAC Settings → Connectivity → Network → IPMI Settings.

Server IPMI settings used for Apache CloudStack Host HA
Enable the BMC/IPMI interface and make sure its management address is reachable from the CloudStack Management Server.
02

Host HA provider

Configure Host HA for the KVM cluster

Open Infrastructure, select your KVM cluster, and open its HA configuration. Choose the KVM Host HA provider for the cluster. The provider performs host health checks and uses neighboring KVM hosts to help determine whether workload disks are still active before CloudStack attempts recovery or fencing.

Why the cluster needs another KVM host

The KVM HA provider relies on another healthy host in the cluster for parts of its investigation process. A single-host cluster cannot provide meaningful host failover for running workloads.

Configure Apache CloudStack KVM Host HA provider
Associate the KVM Host HA provider with the cluster that contains your participating compute hosts.
03

Enable Host HA

Enable HA for the cluster and participating hosts

After selecting the provider, enable Host HA at the cluster level and make sure the intended hosts are included. Host HA is opt-in: a provider must be configured and HA must be enabled before CloudStack manages the hosts through the HA state machine.

Available

Healthy

The host is HA-enabled and currently eligible for HA management.

Ineligible

Check requirements

OOBM, neighboring hosts, host state, or other eligibility requirements may not be satisfied.

Disabled

Not managed

Host HA has not been enabled for this host or its containing partition.

Enable Apache CloudStack Host HA for a KVM cluster
Enable Host HA only after the cluster contains the required hosts, networking, and shared storage.
04

Out-of-band management

Configure OOBM for each KVM host

Open the details page for the first KVM host and configure Out-of-band Management. For an IPMI 2.0 BMC, use CloudStack's IPMITOOL driver and enter the BMC address, port, username, and password. Repeat this configuration for every host that will participate in Host HA.

FieldExamplePurpose
DriverIPMITOOLUses the Management Server's ipmitool binary.
Address192.168.10.11BMC/iDRAC address, not the host OS IP.
Port623Default IPMI port.
UsernamecloudstackBMC account permitted to query and change server power state.
!

Do not enter the KVM host's normal management IP

The OOBM address must point to the hardware management controller. CloudStack needs this separate control path so it can power-cycle or fence a host even when the host operating system is unreachable.

Configure Apache CloudStack out-of-band management using IPMI
Configure the physical server's BMC address and credentials for CloudStack out-of-band management.
05

Enable OOBM

Enable and verify out-of-band power management

After saving the OOBM configuration, enable it for the host. CloudStack must be able to read the server's power state before Host HA can reliably recover or fence that host. Repeat the process for the second KVM host and any additional participating hosts.

If you want to test IPMI independently of CloudStack, run the command from the Management Server. Install ipmitool first if it is not already installed.

Management Serverbash
sudo apt update
sudo apt install -y ipmitool

sudo ipmitool -I lanplus \
  -H 192.168.10.11 \
  -U cloudstack \
  -a chassis power status

The -a option prompts for the BMC password instead of placing it directly in your shell command. Replace the example BMC address and username with the values from your environment.

Enable Apache CloudStack out-of-band management on a host
Enable OOBM and verify that CloudStack can read the physical host's power state.
06

Compute offering

Create a compute offering for HA workloads

Open Service Offerings → Compute Offerings and create a new offering for the workloads that should use CloudStack instance HA. Choose the CPU, memory, CPU speed, and other resource limits that make sense for your lab or application.

FieldExample
NameHA Small
CPU cores2
CPU speed2000 MHz
Memory2048 MB
Create an Apache CloudStack compute offering for HA-enabled instances
Create a compute offering with the CPU and memory resources needed by your HA workload.
07

HA offering

Enable high availability on the compute offering

In the compute-offering form, enable the High Availability option before saving. Instances deployed with this offering will be marked for CloudStack instance HA.

HA-enabled instance shutdown behavior

CloudStack cannot reliably distinguish an unexpected guest shutdown from a shutdown initiated inside the guest operating system. If you want an HA-enabled instance to remain stopped, stop it through the CloudStack UI or API rather than running the guest's own shutdown command.

Enable high availability on an Apache CloudStack compute offering
Enable the High Availability option so instances using this offering are monitored for HA recovery.
08

HA-enabled instance

Deploy an instance with the HA compute offering

Create a new instance using the template prepared in Guide 05. During the compute-offering step, select the HA-enabled offering you just created. Finish the normal deployment workflow and wait for the instance to reach the Running state.

Offering

HA Small

The service offering has High Availability enabled.

Primary storage

Shared NFS

The instance disks remain accessible to another suitable host after failover.

Select an HA-enabled compute offering while creating an Apache CloudStack instance
Select the compute offering that has the High Availability option enabled.
09

Failover test

Test the HA workflow and verify recovery

Confirm which KVM host is currently running the HA-enabled instance. Before testing anything, verify that the second host is Up, has spare CPU and memory capacity, can access the same NFS primary storage, and has working OOBM.

Apache CloudStack HA-enabled instance ready for failover testing
Finish deploying the HA-enabled instance and confirm that it is running normally before simulating a host failure.
!

Perform failure testing only in a controlled lab

Do not abruptly power off a production hypervisor just to verify HA. Use disposable test workloads and a maintenance window, and make sure you can recover the physical host manually if the fencing or power-management configuration is wrong.

In a lab, simulate an unexpected host failure from outside CloudStack—for example by powering off the compute server through its BMC. Then monitor the host HA state, events, alerts, and the HA-enabled instance from the CloudStack UI.

CheckWhat you should see
Host HA stateCloudStack investigates the failed host and progresses through the Host HA recovery/fencing workflow.
OOBM actionsRecovery/fencing operations can use the BMC to reset or power off the failed host when required.
InstanceAfter CloudStack confirms it is safe to do so, the HA-enabled instance can be restarted on another suitable host.
Original hostA host that cannot be recovered may be fenced and placed into maintenance, requiring administrator intervention.

Do not expect recovery to happen at one fixed number of minutes. Host HA performs multiple checks before it decides that a host can be safely recovered or fenced, and the timing depends on the HA configuration and the failure scenario. CloudStack is intentionally conservative because it must avoid starting a second copy of an instance while the original host might still be writing to the same disks.

High availability configured

Your KVM cluster now has an HA recovery path.

You have configured Host HA, out-of-band power management, a shared-storage HA compute offering, and an HA-enabled instance. Continue monitoring host capacity and always leave enough resources on the remaining hosts to absorb workloads when one host fails.