Disaster Recovery Model

The following sections explain the Disaster Recovery model in detail.

Overview

Having a robust Disaster Recovery (DR) plan in place helps you recover from a failure of the production environment and continue using the application. Adeptia recommends you that you have the DR environment ready as a failover if anything unpredictable happens to your application, storage, or the database.

The illustration given below depicts the DR model recommended by Adeptia.

This can be achieved by having a two zone deployment of the application that share the same database for Read/Write purpose. In an ideal condition, all the requests are handled and processed in the production environment. The application in DR environment does not perform any kind of processing, nor does it interact with the database.

The following description gives you an idea about all the components of the DR model and thus the achieved infrastructure.

Region 1 and Region 2 are the physical locations of the Kubernetes Cluster where the production and DR environment are set up respectively.
Production environment governs the Adeptia Connect application that is in action and executes all the requests. It also has a storage and two databases (Primary Database and Sync Database).
DR environment has another copy of the Adeptia Connect application and a storage. The application in the DR environment remains inactive until the application in the production environment goes down and there is a need to switch to the DR environment. This environment is connected to the same database being used in the production environment. However, the application deployed here neither interacts nor stores any data in the database until it gets active and starts processing the requests in the face of a disaster.
Storage in production environment is synced to the storage in the DR environment that comes into picture when the application in the DR environment is in use.
Database is shared by both the environments. This model has a Primary Database synced to a Sync Database. When the Primary Database fails, the Sync Database takes over.

Deploying the application in production and DR environments

The DR model encapsulates the environments – Production and DR – to be set up in Kubernetes Clusters located in two different zones in the cloud.

You can deploy the application and the databases in the production environment by following the standard steps and prerequisites described on this page. Additionally, ensure that,

You have set the required DNS records to point to the production environment so as to send all the requests from an end user to this environment for processing.

As Adeptia recommends that you use two databases (Primary Database and Sync Database) when setting up the DR model, you must use DNS for switching from one database to another.

While deploying the application in DR environment, you need to take the following prerequisites into account in addition to the standard ones prescribed for the regular deployment.

In the values.yaml file, set the value for the environment variable EXECUTE_STATIC_JOB to false so that it doesn't rewrite the static content in the DR storage.
In the listener section of the global values.yaml file, set the value for the environment variable replicaCount to 0 (zero) to avoid processing of data in the DR environment.
In the global values.yaml file, set the value for the environment variable QUARTZ_STATUS_ONBOOTUP to pause. This pauses the Scheduler.
Ensure that the application version is same as that in the production environment.

Upgrading the environments

The prerequisites for upgrading the production and DR environments remain the same as that for the deployment of the application discussed in this section.

Using DR to recover from a failure

Adeptia's DR model covers the following two scenarios and the action steps you need to perform to address the failure and recover the infrastructure, application, and the data with a quick turnaround time.

When the production application goes down

Perform the following steps to switch to the DR environment:

Resume the Scheduler by following the steps given below.
1. Log in to Adeptia Connect.
2. Click Account and select Maintenance.
3. Click Scheduler in the left pane.
4. Click Resume Scheduler.
  The Scheduler status changes to 'Scheduler is running'.
You can also resume the Scheduler by using an API. The format for the API is given below.
```
<Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Resume&page=1&start=0&limit=25
```

Run the command in the following format to scale up the listener pod from 0 to 1.

kubectl scale --replicas=1 deployment <name of the listener deployment> -n <namespace>

Update the DNS records to point to the DR environment.

Switching back to the production environment

Once the production environment is restored, you need to do the followings to switch back to it from the DR environment:

Replicate the data from the DR storage to the production storage.
Update the DNS records to point to the production environment.

When the Primary Database fails

Perform the following step to switch to the Sync Database:

Update the DNS records to point to the Sync Database.

Switching back to the Primary Database

Once the Primary Database is restored, you need to do the followings to switch back to it from the Sync Database:

Ensure that the Primary Database is connected and performing Read/Write operations.
Replicate the data from the Sync Database to the Primary Database.
Update the DNS records to point to the Primary Database.