Data Container Failover


Overview

Private DNS Version 1.1.0 includes a built-in failover mechanism for the data container. In order to utilize this failover mechanism, more than one data container must be created with exact configuration parameters for each container . One is elected as “primary” data container and all others operate as  “replica” data containers. Changes to the primary data container’s data sets are detected by tailing oplogs and changes are streamed to replicas.

Initial Failover Setup

To achieve redundancy and service resiliency, it is highly recommended that the replica data containers should not be on the same host as the primary. Ideally, each data container should be on a separate host so any one replica can assume the role as primary at a moment’s notice.

Starting data containers in the replica role:

Copy the data container config from docker-compose.yml to remote host and bring up container(s) and verify connectivity in your firewall and forwarding setup

All web/xfr/dns/cache containers must have all data containers listed in their configuration

Each data container’s configuration must list the other data containers and not itself

E.g. container A has containers B and C listed; container B has containers A and C listed; and container C has containers A and B listed.

Set standby data containers to standby using the advanced menu accessible typing (?).

Set one data container to the primary role:

Set one data container to primary using the advanced menu accessible typing (?).

Notes:

One data container must be set to primary, otherwise the cluster will not function.

Failover In Action:

  1. If it has not already stopped, bring the primary data container down by stopping it (e.g. docker-compose stop or docker stop). If a container has broken to the point where it can no longer function as a primary, it’s best to destroy it and create a new one.  
  2. Choose one replica data container to promote to primary. View advanced configuration options with (?) hotkey. Switch the Primary/Replica toggle button to Primary mode. Select Commit.
  3. After setting the new primary data container, wait 30 seconds, and restart the API Service on any web containers.

Failback:

New containers:

Go through initial setup process to add a new data container above

If the new data container’s address is different from the old address,

add the new address to the other data containers

Set the new data container to the replica role via the advanced menu accessible at (?)

Make sure the data is synced. 

Set the current primary data container back to the replica role via the advanced menu accessible at (?)

Set the new data container to the primary role via the advanced menu accessible at (?)

After 30 seconds, the API process must be restarted to maintain data integrity.


Existing containers:

After repairing the container, bring it up using Docker-Compose

Set the repaired container to the replica role via the advanced menu accessible at (?)

Make sure the data is synced. 

Set the current primary data container back to the replica role via the advanced menu accessible at (?)

Set the repaired data container to the primary role via the advanced menu accessible at (?)

After 30 seconds, the API process must be restarted to maintain data integrity.

WARNING:

The cluster must not be configured with more than one primary data container. If this is done, there will be data loss and unpredictable behavior.

Request a Demo

Contact Us

Get Pricing