High Availability Settings

About High Availability

High Availability is a redundancy feature included with the Dante Domain Manager (DDM) Platinum license. This feature enables a secondary or backup server (the ‘HA Only node’) to take over if the primary server (the ‘Standalone node’) goes down or goes offline.

In normal operation, the Standalone node’s state is ‘active’, and the HA Only node’s state is ‘auxiliary’.

All configuration data on the active server is dynamically replicated to the auxiliary server. If the auxiliary server detects that the active server is offline, it will take over as the active DDM server and all Dante clients (devices and controllers) will connect to it.

High Availability (HA) allows a DDM system to continue normal operation in the case of a server failure. In a DDM system with or without HA, audio is never disrupted if a server goes down. The benefit of HA is that control connections - i.e. being able to log into Dante Controller and make changes to devices and routes – will resume after a brief disruption, rather than when the server is restored. High Availability requires additional server and network resources.

Users logged into a DDM server in Dante Controller will have to log in again in the event of an active server failure. Device configuration via embedded controllers and Host CPU interfaces may not be possible while the system is in the process of failing over.

Note:  Dante Domain Manger High Availability is not related to Dante device redundancy and setting up a Dante redundant (secondary) network.

How Does it Work?

The DDM high availability implementation requires 3 servers: the Standalone node, the HA Only node, and the arbiter.

The arbiter serves as a tiebreaker in the event the network becomes partitioned – in which case the server which is still in communication with the arbiter takes over as the active server.

If at any point there are not at least two servers visible to each other, the system will switch to ‘read-only’ mode: existing audio subscriptions will be maintained, but configuration changes via the DDM user interface and Dante Controller will be disabled.

High Availability utilizes a virtual IP address. Devices and controllers connect to this virtual address instead of the physical address of the individual servers. The virtual IP address is configured as an additional address on the network interface of the currently active server. In the event the active server becomes disconnected, it gives up this address. The auxiliary server then takes over the virtual address and configures it on its own network interface.

Server Requirements

The Standalone and HA Only nodes should ideally be specified identically, in line with the standard DDM system requirements. If the servers cannot be specified identically, you may encounter performance degradation in the event of a failover.

Standalone and HA Only Server Minimum Requirements:

  • A CPU with a minimum of 2 cores (for VMs spec 2 CPUs)
  • 8GB of RAM.
  • For systems that include more than 200 devices, 16GB of RAM is recommended.
  • At least 20 GB hard drive space

The arbiter server must be reliable, but does not replicate the DDM database, and therefore does not need to match the performance of the Standalone and HA Only nodes.

Arbiter Server Minimum Requirements:

  • A CPU with a minimum of 2 cores (for VMs spec 2 CPUs)
  • 4GB of RAM
  • At least 20 GB hard drive space

Related Topics Link IconRelated Topics

IP Addresses and Hostnames

Each server must be specified with a unique IP address and hostname. All server IP addresses must be in the same IP subnet. Unique hostnames, if required, must be set before enabling HA Mode.

Additionally, a virtual IP address will be required for the cluster. The virtual IP address must be in the same subnet as the DDM servers, currently unused, and not allocated (or enabled for allocation) by DHCP.

A DNS ‘A’ record can be configured as an alias for the Virtual IP address. If present, the ‘A’ record should eventually match the FQDN for the DDM installation, but can be configured to match after forming the cluster in the DDM ‘Network & Security’ settings page.

Network Time

To ensure accurate replication in the event of server clock drift, you can optionally provide the DDM server with access to an NTP server. NTP servers can be specified for the DDM server using the Administration Menu. If your network is not connected to the Internet, specifying an alternative NTP server is a requirement.

Device Discovery

Unicast DNS is strongly recommended for discovery in HA mode.

Dante Discovery Service (mDNS) is not supported in HA mode, and will be deactivated automatically when HA is enabled.

For unicast DNS, SRV records allow devices/controllers to discover the DDM Server Cluster. The SRV records in your DNS server should point to the cluster’s virtual IP address, or FQDN (if you are using a hostname for the virtual IP address).

Note:  If you have previously enrolled devices into a different DDM server or a DDM server in Standalone mode, they may experience discovery issues. See 'Transitioning to HA mode from Standalone Mode'.

It is not recommended to use manual device enrollment by IP address with HA mode - but if it is unavoidable, the following should be considered:

  • The HA Cluster must be configured before enrolling any devices into a domain. When you manually enroll a device by IP address into a domain, the DDM server sends a static reference to the device, based on the currently configured FQDN for the DDM. The static reference is the DDM server’s IP address. If the devices are referencing the IP address of the standalone node (active) server rather than the Virtual IP address of the cluster when the server fails over to the HA Only node (auxiliary) server, all devices will be shown as offline and will not appear in Dante Controller.

Note:  Additional precaution should be taken when manually enrolling devices by IP address with Dante firmware versions 4.0-4.2 as when a device is manually enrolled by IP, discovery will be permanently disabled in the device until the device has been reset and domain credentials cleared. This results in devices becoming undiscoverable if you wish to use DNS for discovery in the future. (This issue should be resolved in future Dante device firmware versions)

Licensing

Only one DDM License (Platinum) and product key are required to license (activate) the HA Cluster.

When configuring and activating the severs, you will need to enter the product key on the Standalone node and the HA Only node. Internet access is required to activate these servers. The Arbiter server does not need to be licensed.

Prior to installation and activating, it must be decided which server will be the Standalone node (typically the preferred active server). The other becomes the HA Only Node and inherits its license state dynamically from the Standalone node.

Important:  If you experienced a failover to the HA Only node, it is important to make sure the Standalone node becomes the active server again when it is restored. If the HA Only node remains the active server for more than 30 days continuously, the license on the HA Only node will become deactivated. To avoid license deactivation when the Standalone node is restored, change it to the active Server. If the Standalone node is unrecoverable, you can make the HA Only node a Standalone node.

Setting Up and Configuring HA

Important:  The HA Cluster should be set up and configured before enrolling devices into domains.

Installing and Activation

  1. Begin by installing the DDM ISO file onto 3 virtual machines or bare metal servers.
  2. Boot up your preferred active server, open the DDM web interface for this server and follow the prompts to install DDM as a ‘Fresh Installation.’ In the context of HA, this is the Standalone node. An Internet connection is required for activating the Standalone and HA Only nodes.
  3. Configure a TLS certificate, if required (note that the TLS certificate will be shared by all nodes). The TLS certificate should contain the subject name for the FQDN for the virtual IP address.
  4. Boot up your preferred auxiliary (backup) server.
  5. Open the DDM web interface for this server.
  6. On the ‘Installation & Configuration’ page, choose ‘High Availability Redundant Node’ and click Next.
  7. Follow the prompts to complete the installation, using the same product key you used for the Standalone node. In the context of HA, this is an HA Only node.
  8. Start the DDM appliance on your preferred arbiter server.
  9. Open the DDM web interface for this server.
  10. On the ‘Installation & Configuration’ page, choose ‘High Availability Arbiter’ and click Next.
  11. Follow the prompts to complete the installation. You do not need a product key to install an arbiter node. This server becomes the Arbiter node.

Enabling HA Mode

  1. On your Standalone node, navigate to Settings > High Availability.
  2. In the ‘Node Status’ section, copy the Security Key.
  3. In a new browser tab, navigate to the URL or IP address of the HA Only Node.
  4. In the ‘Node Status’ section, click Edit and paste in the security key.
  5. Repeat the two steps above for the Arbiter node.
  6. Return to the browser tab for the Standalone node, and click the toggle switch to enable HA mode.
  7. In the Cluster Settings, enter a virtual hostname or IP address. The virtual hostname and IP address of the cluster must be unique on the network, and in the same subnet as the DDM servers.
  8. In the ‘Node 1’ field, enter the hostname or IP address of the Standalone node.
  9. In the ‘Node 2’ field, enter the hostname or IP address of the HA Only node.
  10. In the ‘Arbiter’ field, enter the hostname or IP address of the Arbiter node.
  11. Click Save Changes to enable the cluster.

Note:  While the cluster is active, you cannot use the DDM UI on the auxiliary and arbiter servers.

Changing the Active Server

To change the active server to auxiliary and the auxiliary server to active, in the High Availability settings for the cluster or active server, click Change Active.

Making the HA Only Node a Standalone Node / Active Server

If the original Standalone node goes offline and is unrecoverable and the HA Only node is now the active server, you can upgrade the HA Only node to a Standalone node.

To make an HA Only node Standalone, in the DDM UI navigate to Settings > High Availability and click Make Standalone.

The license for the new Standalone mode must then be deactivated and reactivated as a standalone license (requires Internet access). It is likely you will need to contact Audinate Support to reset the license, so you can activate the HA Only node as Standalone. Therefore, it is advisable to wait until you’ve been in contact with Audinate support to reset your license before transitioning to a Standalone Node (you can keep the HA Only node as the Active server in HA Mode for 30 days before the license becomes deactivated.)

Transitioning To / From HA

Disbanding the HA Cluster

To disband the HA cluster, on the active server, go to Settings > High Availability and click Disband.

Transitioning from HA Mode to Standalone Mode

Because devices remember the IP address or hostname of the DDM server, the best way to avoid device discovery issues when transitioning from a system set up in HA Mode to Standalone mode is to give the Standalone node the IP address and hostname of the former cluster.

Transitioning to HA Mode from Standalone Mode

The best way to avoid device discovery issues when transitioning from a system already setup in Standalone mode to HA mode is to give the Standalone node a new IP address and hostname, and give the cluster the former Standalone node’s IP address/hostname.

Resolving Issues with Device Discovery

If a device that has previously been enrolled in a domain is not discovering the DDM, you can reset the device and clear the domain credentials using Dante Controller.

  1. Isolate the device from the rest of the Dante network and the DDM server.
  2. Disconnect and reconnect the device.
  3. When the device appears in Dante Controller, double-click it to open the Device View for the device.
  4. From the Device menu, select Clear Domain Credentials.
  5. In the popup window, click the Clear Config button.

Updating DDM in HA Mode

It is advisable to make a backup of the DDM system configuration before updating.

To update DDM in HA mode:

  1. Disband the HA cluster.
  2. Update each server independently.
  3. Recreate the cluster.

Note:  While the cluster is disbanded, devices will present as offline (because the virtual IP address/hostname is temporarily not attached to any network interface).