Page Comparison

The following sections explain the Disaster Recovery model in detail.

...

1. Scale down the Runtime and Listener deployments in the PROD environment.
   To scale down the Runtime deployments, use the following format:

   Code Block
   language css theme Midnight
   kubectl scale --replicas=0 deployment <Release Name>-ac-runtime -n <namespace>

   To scale down the Listener deployments, use the following format:

   Code Block
   language css theme Midnight
   kubectl scale --replicas=0 deployment <Release Name>-ac-listener -n <namespace>

   
   

2. Pause Scheduler in the PROD environment.

   Code Block
   language css theme Midnight
   https://<webapp_Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Pause&page=1&start=0&limit=25

   
   

3. Scale down Rabbit MQ statefulset in the PROD environment.

   Code Block
   language css theme Midnight
   kubectl scale --replicas=0 statefulset <Release Name>-ac-rabbitmq -n <namespace>

   
   

4. Scale down Rabbit MQ statefulset in the DR environment.


   Code Block
   language css theme Midnight
   kubectl scale --replicas=0 statefulset <Release Name>-ac-rabbitmq -n <namespace>

   
   

5. Initiate failover for the storage account using the following steps:

   Info
   >> Initiating failover is required only when the Azure File Share is not accessible due to any failure. >> As per Microsoft documentation, the time it takes to failover after initiation can vary though typically less than one hour.

   1. Login into your Azure Portal.

   2. Go to the Storage Account being used in the PVC of Prod environment.

   3. Within that storage account go to Data Management > Redundancy section.

   4. Click Prepare for failover and follow the instruction given on the screen. The failover process starts.

   5. Once the Account Failover process completes the data from secondary region will be accessible from the existing endpoint of the storage account.

6. Copy the web/repository folder from PROD to DR environment.

   Code Block
   language css theme Midnight
   ./azcopy sync "https://[account].file.core.windows.net/[PVC]/web/repository/?[SAS]" "https://[account].file.core.windows.net/[PVC]/web/repository/?[SAS]" --recursive=true

   For example:
./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/web/repository/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/web/repository/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true

7. Copy the recovery folder from PROD to DR environment.

   Code Block
   language css theme Midnight
   ./azcopy sync "https://[account].file.core.windows.net/[PVC]/recovery/?[SAS]" "https://[account].file.core.windows.net/[PVC]/recovery/?[SAS]" --recursive=true

   For example:
   ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/recovery/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/recovery/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true

8. Copy the reprocessing folder from PROD to DR environment.

   Code Block
   language css theme Midnight
   ./azcopy sync "https://[account].file.core.windows.net/[PVC]/reprocessing/?[SAS]" "https://[account].file.core.windows.net/[PVC]/reprocessing/?[SAS]" --recursive=true

   For example:
   ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/web/reprocessing/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/reprocessing/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true

9. Copy the Rabbitmq PVC content from PROD to DR environment.

   Code Block
   language css theme Midnight
   ./azcopy sync "https://[account].file.core.windows.net/[PVC]/?[SAS]" "https://[account].file.core.windows.net/[PVC]/?[SAS]" --recursive=true --mirror-mode=true

   For example:
   ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-ef345e9d-xxxx-xxxx-xxxx-54e38de9cbd1/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-4a47fa22-xxxx-xxxx-xxxx-f6538dfff1a7/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true --mirror-mode=true

10. Scale up RabbitMQ statefulset in the DR environment.

    Code Block
    language css theme Midnight
    kubectl scale --replicas=1 statefulset <Release Name>-ac-rabbitmq -n <namespace>

    
    

11. Scale up Listener in the DR environment.

    Code Block
    language css theme Midnight
    kubectl scale --replicas=1 deployment <Release Name>-ac-runtime -n <namespace>

    
    

12. Resume Scheduler in the DR environment.
    https://<webapp_Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Resume&page=1&start=0&limit=25
13. Update DNS record to route all request the requests to the DR environment.

Switching back from DR to PROD Environment

1. Scale down Runtime and Listener deployments in the DR environment.

2. Pause Scheduler in the DR environment.

3. Scale down Rabbit MQ statefulset in the DR environment.

4. Scale down Rabbit MQ statefulset in the PROD environment.

5. Copy web/repository folder from DR to PROD environment.

6. Copy recovery folder from DR to PROD environment.

7. Copy reprocessing folder from DR to PROD environment.

8. Copy Rabbitmq PVC content from DR to PROD environment.

9. Scale up RabbitMQ statefulset in the PROD environment.

10. Scale up Listener in the PROD environment.

11. Resume Scheduler in the PROD environment.

12. Update DNS record to route all request the requests to the PROD environment.

Using DR to recover from a failure

...

1. Log in to your Azure Portal.

2. Go to the Storage Account being used.

3. Initiate “Account Failover”. The failover process starts.

4. Once the Account Failover process completes the data from secondary region will be accessible from the existing endpoint of the storage account.

Appendix

...

: Getting the Shared Access Signature (SAS) of your Azure Storage Account

1. Login into Log in to your Azure Portalportal.

2. Go to the Storage Account.

3. Within In the Storage account Account, go to Security + Networking > Shared Access Signature.
   

   

4. Enable Grant the required permission permissions as show below:
   

5. Update the expiry date/time as per your need.

6. In the Allowed Protocol enable HTTP and HTTPS bothprotocols field, select HTTPS and HTTP.

7. Click Generate SAS and Connection String as shown belowconnection string:
   

8. Use the File Service service SAS URL.