The following sections explain the Disaster Recovery model in detail.
...
Scale down the Runtime and Listener deployments in the PROD environment by running two commands in the following formats.
Code Block language css theme Midnight kubectl scale --replicas=0 deployment <Release Name>-ac-runtime -n <namespace>
Code Block language css theme Midnight kubectl scale --replicas=0 deployment <Release Name>-ac-listener -n <namespace>
Pause Scheduler in the PROD environment.
Code Block language css theme Midnight https://<webapp_Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Pause&page=1&start=0&limit=25
Scale down Rabbit MQ statefulset in the PROD environment.
Code Block language css theme Midnight kubectl scale --replicas=0 statefulset <Release Name>-ac-rabbitmq -n <namespace>
Scale down Rabbit MQ statefulset in the DR environment
Code Block language css theme Midnight kubectl scale --replicas=0 statefulset <Release Name>-ac-rabbitmq -n <namespace>
Initiate Failover failover for the storage account using below the following steps:
Info >> These failover steps are not mandatory. These steps are mandatory Initiating failover is required only when the Azure File Share is not accessible due to any failure.
>> As per Microsoft guidelineguidelines, the time it takes to failover after initiation can vary though typically less than one hour.
Login into your Azure Portal.
Go to the Storage Account being used in the PVC of Prod environment.
Within that storage account go to Data Management > Redundancy section.
Click Prepare for failover and follow the instruction given on the screen. The failover process starts.
Once the Account Failover process completes the data from secondary region will be accessible from the existing end point endpoint of the storage account.
Copy the web/repository folder from PROD to DR environment.
Code Block language css theme Midnight ./azcopy sync "https://[account].file.core.windows.net/[PVC]/web/repository/?[SAS]" "https://[account].file.core.windows.net/[PVC]/web/repository/?[SAS]" --recursive=true
For
Exampleexample:
./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/web/repository/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/web/repository/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true
Copy the recovery folder from PROD to DR environment.
Code Block language css theme Midnight ./azcopy sync "https://[account].file.core.windows.net/[PVC]/recovery/?[SAS]" "https://[account].file.core.windows.net/[PVC]/recovery/?[SAS]" --recursive=true
For
Exampleexample:
./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/recovery/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/recovery/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true
Copy the reprocessing folder from PROD to DR environment
Code Block language css theme Midnight ./azcopy sync "https://[account].file.core.windows.net/[PVC]/reprocessing/?[SAS]" "https://[account].file.core.windows.net/[PVC]/reprocessing/?[SAS]" --recursive=true
For Example:
./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/web/reprocessing/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/reprocessing/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=trueCopy the Rabbitmq PVC content from PROD to DR environment
Code Block language css theme Midnight ./azcopy sync "https://[account].file.core.windows.net/[PVC]/?[SAS]" "https://[account].file.core.windows.net/[PVC]/?[SAS]" --recursive=true --mirror-mode=true
For Example:
./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-ef345e9d-xxxx-xxxx-xxxx-54e38de9cbd1/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-4a47fa22-xxxx-xxxx-xxxx-f6538dfff1a7/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true --mirror-mode=trueScale up RabbitMQ statefulset in the DR environment.
Code Block language css theme Midnight kubectl scale --replicas=1 statefulset <Release Name>-ac-rabbitmq -n <namespace>
Scale up Listener in the DR environment.
Code Block language css theme Midnight kubectl scale --replicas=1 deployment <Release Name>-ac-runtime -n <namespace>
- Resume Scheduler in the DR environment.
https://<webapp_Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Resume&page=1&start=0&limit=25 - Update DNS record to route all request to DR environment
...
Follow the steps below to point to secondary storage location:
Login into Log in to your Azure Portal.
Go to the Storage Account being used.
Initiate “Account Failover”. The failover process starts.
Once the Account Failover process completes the data from secondary region will be accessible from the existing end point endpoint of the storage account.
Appendix A: Getting Shared Access Signature (SAS) of your Azure Storage Account
...