You can configure two appliances as an HA (high availability) pair to provide hardware redundancy for core network services and Infoblox Advanced DNS Protection. For more information, see About Infoblox Advanced DNS Protection. An HA pair can be a Grid Master, a Grid Master candidate, a Grid member, or an independent appliance. An HA pair can also comprise a physical appliance and a virtual appliance, two physical appliances, or two virtual appliances. The two nodes that form an HA pair—identified as Node 1 and Node 2—are in an active/passive configuration. The active node receives, processes, and responds to all service requests. The passive node constantly keeps its database synchronized with that of the active node, so it can take over services if a failover occurs. A failover is the reversal of the active/passive roles of each node; that is, when a failover occurs, the previously active node becomes passive and the previously passive node becomes active. You can configure an HA pair in either IPv4, IPv6, or in dual mode. An IPv4 HA pair uses IPv4 as the communication protocol between the two nodes and an IPv6 HA pair uses IPv6 as the communication protocol between the two nodes. But in a dual mode HA pair, you can select either IPv4 or IPv6 as the communication protocol between the two nodes. Note that when you add a dual mode HA member to a Grid, the communication protocol between the two nodes of an HA pair must the same as the Grid communication protocol.

When you configure an HA pair using the IB-4030-10GE (Rev-1 or Rev-2) appliance for DNS cache acceleration, the passive node does not operate with a pre-loaded cache or hot cache during a failover; it builds up the DNS cache over time. For more information about HA and other limitations for the IB-4030-10GE appliances, refer to the Infoblox DNS Cache Acceleration Application Guide. For Infoblox , only the active node in an HA pair handles DNS traffic. The passive node is in a standby mode ready to take over if a failover occurs.
The appliance uses the following components in the HA functionality:

• bloxSYNC: An Infoblox proprietary mechanism for secure, real-time synchronization of the database that maintains the data, system configuration, and protocol service configuration between the two nodes. With bloxSYNC, the nodes continuously synchronize changes of their configurations and states. When a failover occurs, the passive node can quickly take over services. For information, see About HA Failover below.

• VRRP (Virtual Router Redundancy Protocol): An industry-standard, MAC-level HA failover mechanism. VRRP utilizes the concept of an active and passive node that share a single VIP (virtual IP) address. When the active node that owns the VIP becomes unavailable, the passive node takes over the VIP and provides network core services. For information about VRRP, refer to RFC3768,Virtual Router Redundancy Protocol (VRRP) and see VRRP Advertisements below.

Using bloxSYNC and VRRP combined, if the active node fails or is taken offline for maintenance purposes, the passive node assumes the VIP and continues to respond to requests and services with minimal interruption. You can deploy an HA pair as a Grid Master, a Grid member, or an independent HA. To deploy an independent HA pair, see Deploying an Independent HA Pair. To deploy an HA Grid Master, see Creating a Grid Master.

Planning for an HA Pair

To achieve high availability, the HA and LAN1 (or VLAN) ports on both the active and passive nodes are connected to switches on the same network or VLAN. Both nodes in an HA pair share a single VIP address and a virtual MAC address so they can appear as a single entity on the network. You can also assign IPv6 addresses for each of the active and passive nodes, in addition to the IPv6 VIP address.

In HA,  each node must configure two addresses: the VRRP public address on the LAN1 interface and the VRRP HA address on the HA interface. An HA pair consists of a set of five IP addresses, all of which must belong to the same subnet. Each device in an HA pair joins the multicast address on both the HA and public interfaces.

As illustrated in the following figure, the VIP and virtual MAC addresses link to the HA port on each node. Select five IP addresses on the same network before you configure an HA pair, as follows:

• VIP: For core network services and for management purposes when the MGMT port is disabled. Both nodes share the same VIP. The VIP is the true public address in which services and daemons are active.

• Node 1 HA (active): Source IP for the VIP and VRRP advertisements. Listens on both its LAN and HA ports. For an active HA node, both the LAN interface/address and the HA interface/address belong to the VRRP multicast group.

• Node 1 LAN1 (active): For management through SSHv2 and listens for VRRP advertisements from the HA port.

• Node 2 HA (passive): Listens for VRRP advertisements on the LAN port. For a passive HA node, only the LAN interface/address belongs to the VRRP multicast group (using the LAN port's MAC address).

• Node 2 LAN1 (passive): Source IP for SSL VPN to the VIP of the active node and receives bloxSYNC from the VIP.

The above configuration holds good only for IPv4 VRRP configurations. IPv6 VRRP configurations require only three addresses: the VIP and the LAN1 interfaces. For the IPv6 dedicated HA interfaces, NIOS uses the link local IPv6 address which you do not need to provide.
HA Pair

When you deploy a vNIOS HA pair, ensure that the port connection allows for more than one MAC address per vNIC. For example, if you deploy a vNIOS HA pair in VMware vSphere, the port-profile to which the vNIOS HA and LAN ports connect should allow for more than one MAC address per vNIC. You can do this by changing the security settings of the port-group to accept "MAC address changes" and "Forged transmits," as illustrated in the following figure.
Configuring port-profile in VMware vSphere

Limitations of Using a Combination of a Physical Appliance and a Virtual Appliance for HA

Although you can use a combination of a physical and virtual appliance for an HA pair, using the deployment has the following limitations:

• Check whether the NIOS licenses that you subscribed to support both physical and virtual appliances.

• Ensure that the same licenses are installed on both the physical appliance and the virtual appliance.

• You cannot have tagged and untagged interfaces on the same subnet on VMware ESXi hypervisors.

• Virtual appliances do not support tagging.

• LOM (Lights Out Management) is not supported in a hybrid HA setup.

• DSCP (Differentiated Services Code Point) services are not supported on virtual appliances. Therefore, you cannot configure the DSCP value in an HA setup.

• Because port settings are not available for virtual appliances, you cannot join a node if the port settings are overridden.

• You cannot combine a platform on which Advanced DNS Protection hardware is running with a platform on which Advanced DNS Protection Software is running.  

• You cannot configure MTU (Maximum Transmission Unit) in a hybrid HA setup.

• You cannot have a combination of an IB-FLEX and a non IB-FLEX appliance.

• Auto-provisioning is not supported on virtual appliances; therefore, you cannot use the auto-provisioning feature in a hybrid HA setup.

• A hybrid HA setup may cause some performance impact because hybrid HA performance depends on many factors such as the hardware on which the VM is running, the number of VMs contending for the same CPU, RAM, input/output resources, and the overhead generated by the virtualisation layer. 

• Minor performance differences are expected between the two nodes of a hybrid HA pair. Hybrid HA performance may vary, and it depends on the hardware components on which different virtualization platforms are running and the performance delivered by Infoblox hardware appliances. Different use cases will produce different numbers (slightly increased or decreased CPU usage, disk access time, and so on). Such performance variation is expected and is not a cause of concern.

About HA Failover

The appliance supports HA through bloxHA™, which provides a robust failover mechanism. As described in Planning for an HA Pair, both nodes in an HA pair share a single VIP address and a virtual MAC address. The node that is currently active is the one whose HA port owns the VIP address and virtual MAC address. When a failover occurs, these addresses shift from the HA port of the previous active node to the HA port of the new active node, as illustrated in the figure below. 

VIP Address and Virtual MAC Address and HA Failover

Enabling ARP on the Passive Node of an HA Pair

You can enable ARP (Address Resolution Protocol) on the passive node of an HA pair and monitor its status externally. For example, when the active node of an HA pair fails over to the passive node, you can ping the passive node from an external location and monitor its status. By default, ARP is disabled on the passive node of an HA pair. ARP settings on an HA member are preserved during a system restart or reboot, HA switch over, and upgrade. In addition, you do not need to restart the appliance when you modify ARP settings. When the active node becomes passive during an HA failover, ARP on an HA member inherits the settings configured in the database.

You can view detailed status for both nodes of an HA pair through the Detailed Status panel. To view the Detailed Status panel, from the Grid tab, select the Grid Manager tab -> Members tab -> Grid_member checkbox -> Detailed Status icon in the horizontal navigation bar. In the Detailed Status panel, you can view ARP connectivity status for the passive node of an HA pair (Green = The passive HA node is connected to the local router; Yellow = The passive HA node fails to connect to the local router; Gray = ARP is disabled on the passive node of an HA pair). The passive HA node uses arping to test the ARP connectivity with the local router. If the local router is not configured, you may see false warnings even if the ARP connectivity is fine. In case of an ARP connectivity failure, the appliance sends an SNMP trap and an email notification, if configured.
Note that the ARP setting is not preserved on a passive HA node when you reset the appliance using the CLI command
reset all or reset the database using the CLI command reset database.

To enable ARP on an HA passive node:

1. From the Grid tab, select the Grid Manager tab -> Members tab.

2. Select an HA member and click the Edit icon.

3. In the Grid Member Properties editor, select the Network tab -> Advanced tab and complete the following:

   • Enable ARP on HA Passive Node?: Select one of the following:

     • Disable (default): Select this to disable ARP on an HA passive node. This is selected by default.

     • Enable (not recommended): Select this to enable ARP on an HA passive node.

4. Save the configuration and click Restart if it appears at the top of the screen.

HA failover on DNS Nameservers

When an HA failover occurs on NIOS, there is an approximate 4-5 second time interval in which the network is adjusted for the new active node and the new passive node. During this failover period, the active node becomes unresponsive. After the new active node comes up on the network,  the DNS service loads all Response Policy Zone (RPZ) files if RPZ is configured. The larger the RPZ files, the longer it takes to load them, and the longer it takes the DNS service to start serving DNS. For example, on a TE-1425 with RPZs that contain 15 million resource records, it can take approximately one and a half minutes to start serving DNS.

If your nameserver uses Grid replication to keep internal zones up to date and is not configured to use RPZ, then the delay before the DNS service starts serving DNS is slightly longer that it is for the HA failover itself. 

VRRP Advertisements

VRRP advertisements are periodic announcements of the availability of the HA node linked to the VIP. The two nodes in an HA pair include a VRID (virtual router ID) in all VRRP advertisements and use it to recognize VRRP advertisements intended for themselves. Only another appliance on the same subnet configured to use the same VRID responds to the announcements. The active node in an HA pair sends advertisements as multicast datagrams every second. It sends them from its HA port using the source IP address of the HA port (not from the VIP address) and the source MAC address 00:00:5e:00:01:vrrp_id. The last two hexadecimal numbers in the source MAC address indicate the VRID number for this HA pair. For example, if the VRID number is 143, then the source MAC address is 00:00:5e:00:01:8f (8f in hexadecimal notation = 143 in decimal notation).
The destination MAC and IP addresses for all VRRP advertisements are 00:00:5e:00:01:12 and 224.0.0.18 (00:00:5e:00:02:12 and FF02::12 for IPv6 only configurations). Because a VRRP advertisement is a multicast datagram that can only be sent within the immediate logical broadcast domain, the nodes in an HA pair must be in the same subnet together.
As illustrated in the figure below, when you configure an HA pair, only the appliance configured to listen for VRRP advertisements with the same VRID number processes the datagrams, while all other appliances ignore them. The passive node in an Infoblox HA pair listens for these on its HA port and the active node listens on its LAN1 or LAN1 (VLAN) port. If the passive node does not receive three consecutive advertisements or if it receives an advertisement with the priority set to 0 (which occurs when you manually perform a forced failover or request the active node to restart, reboot, or shut down), it changes to the active state and assumes ownership of the VIP address and virtual MAC address.
If both nodes go offline, the one that comes online first becomes the active node. If they come online simultaneously, or if they enter a dual-active state—that is, a condition arises in which both appliances assume an active role and send VRRP advertisements, possibly because of network issues—then the appliance with the numerically higher VRRP priority becomes the active node. The priority is based on system status and events.
If both nodes have the same priority, then the appliance whose HA port has a numerically higher IP address becomes the active node. For example, if the IP address of the HA port on Node 1 is 10.1.1.80 and the IP address of the HA port on Node 2 is 10.1.1.20, then Node 1 becomes the active node.
For more information about VRRP, see RFC 3768, Virtual Router Redundancy Protocol (VRRP).

VRRP Advertisements with a Unique VRID