Setting Up Redundancy
Enabling Redundancy
Before You Set Up Redundancy
In redundancy, both nodes will share the exact same configuration state. When a Backup node connects to a Master node, the Backup will attempt to synchronize itself with the Master. Therefore, before you set up for redundancy the following should be considered:
Start with a fresh install for the Backup node. Because the current configuration of the Backup node will be overwritten, make sure that it does not contain anything valuable. It is a good idea to export any projects that are unique to the Backup before enabling redundancy.
All system configurations relative to the Master node must also resolve on the Backup node. For example, OPC-UA connections and database connections must use addresses that resolve from both nodes, or any OPC-COM servers must be installed and configured identically on both nodes. This means using "localhost" in any of the database connections won't work. You should use the IP Address of the computer instead.
Configure firewalls between the redundancy nodes. Redundant systems need TCP connectivity between each other on a variety of ports. Turning off software firewalls or adding special exception rules for each others' addresses is required. Specifically, the Master node must be able to receive data on TCP/IP port 8750 (you can change this in the Settings), and the Backup node must be able to send outgoing data on that port.
While the OS platform (i.e. Windows, OS X, Linux) for the Master and Backup can differ, it is recommended to have similar OS platforms. If the OS platforms do differ, the Windows machine should be the master system or else the Force Failover option will not work.
However, different versions of the same operating system such as Windows 10 and Windows 8 or OSX 10 and OSX 11 have full functionality.
To Set Up Redundancy
Once the above points are done, setting up redundancy is fairly simple:
On the Master Gateway
Go to the Configure section of the Master Gateway webpage.
Select Systems > Redundancy.
The Redundancy and Network Configuration page is displayed showing different sections and settings.Change the following Settings:
- Redundancy Settings, set Mode to Master
- Backup Mode Settings, type in the Master Node Address (for example, 10.20.8.12)
Click Save Changes. Confirm change to Redundancy Settings page is displayed.
Click Confirm to apply your settings.
Go to the Configure tab and select Systems > Redundancy to ensure the redundancy mode and state is properly set.
On the Backup Gateway
Do the exact same steps 1-6 above on the Backup Gateway webpage, except in step 3 set the Redundancy Settings Mode to Backup.
Verify the correct port setting in Network Settings.
To verify the redundancy setup, that is, to ensure the Master and the Backup Gateways are connected, go the Status tab of the Gateway webpage and click on Systems > Redundancy. The Redundancy page will show the connected nodes and their current states.
After saving, the system connects to the Master and downloads a system Backup, and restarts. Once the restart is complete, the Backup node is synchronized and in communication with the Master.
Redundancy Settings
All redundancy settings are configured in the Ignition configuration Gateway under the Configure tab, Systems > Redundancy. Most settings are used by both the Master and Backup nodes, with their individual settings broken out into separate categories.
It is important to know that while the full system configuration is shared between nodes, redundancy settings are not shared between nodes. Therefore, it is perfectly acceptable to have different values for the same settings on the two nodes. For example, it is possible to have a different Standby Activity Level on both nodes, and, of course, the network settings will often be different.
The Master node shares all configuration with the Backup node, and this means that changes cannot be made to your project from the Backup. In fact, the Designer can never be opened from a Backup node, even if the Master is currently offline.
Redundancy Settings
| Setting | Description |
|---|---|
| Mode | Independent - Redundancy is not enabled and this Ignition system runs as an independent node. Master - This is the Master node, who listens for a connection from the Backup node, and is in charge of managing system synchronization. Backup - This is the Backup node, who will connect to the Master and receive system updates. |
| Standby Activity Level | How the node operates when it is not the Active node. Cold - Performs minimal activities, does not connect to devices, and so on. The purpose is to minimize the load on the network and on devices. Warm - Connect to devices, subscribe to tags and set up all executing objects. The purpose is to minimize fail-over time. |
| Fail-over Timeout | The time, in milliseconds, before the opposite node is determined to be unavailable and this node takes over. |
| Startup Connection Allowance | The time, in milliseconds, to wait on initial startup for a connection to be established before making a decision on the node's activity level. This is used to prevent unnecessary switch over caused by a node starting as active, only to connect and find that the other node is active, resulting in one of the nodes being de-activated. It is important to note that this setting can interfere with the Master Recovery Mode - if the Master is active, it will always request the Backup to de-activate. If this setting is low, or 0, the Master will always become active before connecting to the Backup, and thus "manual recovery" will not be possible. |
Network Settings
| Setting | Description |
|---|---|
| Port | For the Master, the port to listen on. For the Backup, the port to connect to on the Master. |
| Auto-detect Network Interface | If true, the system will automatically select which network interface to use on the machine. If false, the system will bind itself to the interface of the specified address. |
| Network Bind Interface | The address to bind to if Auto-detect is false. |
| Auto-detect HTTP Address | When clients are launched, they are provided with a list of addresses that they may connect to. If this option is true, the list will be generated automatically. If false, they will be provided with the list specified. |
| HTTP Addresses | The list of addresses to give to the clients if auto-detect is turned off. These are the addresses that the clients will attempt to connect to, so the HTTP and HTTPS ports must match the configuration of the system in the Gateway Control Utility. Each node should only list addresses that are for that node (this property on the master should only list possible addresses for the master node and the backup should only list possible addresses for the backup node). |
Master Node Settings
| Setting | Description |
|---|---|
| Recovery Mode | How the Master acts when it connects to a Backup node that is currently active. Automatic - The Master automatically takes back responsibility, and becomes active. The Backup node goes to standby. Manual - The Backup node is allowed to stay active. The Master will become active if the Backup node fails, or if the user requests a switchover from the Gateway configuration page. |
| Runtime Update Buffer Size | How many "runtime status updates" can be queued up in memory before the system stops tracking them and forces a full refresh. These updates represent information that the other node should have in order to have the same running state as the Master when it's forced to take over. This is most often the values of static tags and the current alarm state. Given that the update buffer is only used once the nodes are connected, the default value is usually fine, and only needs to be increased on systems that may have many alarms that change together, or many static tag writes. |
| Enable Connection Whitelist | If true, connections will only be accepted if they originate from an address in the whitelist in the next field below. |
| Connection Whitelist | A list of ip addresses, one per line, to accept connections from. Would normally only have one entry for the backup node, unless the backup machine connects through various network interfaces. |
| Config Update Queue Size | New in 7.9.17 The maximum size (in megabytes) of config updates allowed before a full transfer is performed. |
Backup Node Settings
| Setting | Description |
|---|---|
| Master Node Address | The address where the Master is located. |
| Ping Rate | The time, in milliseconds, between messages from the Backup to the Master. |
| Reconnect Period | How often in millisecond, to re-attempt connection when the Backup node is not connected to the Master. |
| History Mode | How history is treated by the Backup system. If Full, history will be stored normally, as it would be on the Master system. If Partial, history will be cached until the Master is available again and the Backup node is able to determine the exact time that the Master was down. |
Troubleshooting
Redundancy Connectivity
When the two redundant nodes are connected, you will be able to see their state details in the Status section of the Gateway webpage. There are also various other places where the redundancy state is shown as connected.
If the two nodes cannot connect, check the following:
- Verify that the Master address is correct in the Backup. Try to ping the Master machine from the Backup machine, and verify that you're using the correct address for the network card that the Master is connected through.
- If using system names (or domain names), verify that the name is resolving to the correct address by performing a ping.
- Verify that the firewall on the Master is set to allow TCP traffic to the designated port.
- Verify that the Backup is not connecting and then immediately disconnected for some reason.
- Viewing the error log in the gateway console section should show this. If errors are occurring at regular intervals, look at the message for an indication of what is happening. An example of a potential problem is when the failover time is set too low for the given network, which results in many socket read timeout exceptions, which in turn leads to many disconnect or reconnect attempts.
- If errors are occurring, but the cause isn't clear, contact Inductive Automation Support.
Advanced Troubleshooting
A variety of loggers can be found under the Gateway console section by going to "Levels" and searching for "Redundancy". By setting these loggers to a finer level, more information will be logged to the console. This is generally only useful under the guidance of Inductive Automation support personnel, though more advanced users may find the additional logged information helpful.