Versions Compared

Old Version 9

changes.mady.by.user Ashhad Alam

Saved on

compared with

New Version 10

changes.mady.by.user Ashhad Alam

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

While deploying the application in DR environment, you need to take the following prerequisites into account in addition to the standard ones prescribed for the regular deployment. 

  • In the values.yaml file, set the value for the environment variable EXECUTE_STATIC_JOB to false so that it doesn't rewrite the static content in the DR storage.In the listener section of the global values.yaml file, set the value for the environment variable replicaCount to 0 (zero) to avoid processing of data in the DR environment. 

  • In the global values.yaml file, set the value for the environment variable QUARTZ_STATUS_ONBOOTUP to pause. This pauses the Scheduler. 
  • Ensure that the application version is same as that in the production environment.

...

  1. Scale down the Runtime and Listener deployments in the PROD environment by running two commands in the following formats.

    Code Block
    languagecss
    themeMidnight
    kubectl scale --replicas=0 deployment <Release Name>-ac-runtime -n <namespace>


    Code Block
    languagecss
    themeMidnight
    kubectl scale --replicas=0 deployment <Release Name>-ac-listener -n <namespace>


  2. Pause Scheduler in the PROD environment

    https://<webapp_Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Pause&page=1&start=0&limit=25

  3. Scale down Rabbit MQ statefulset in the PROD environment
    kubectl scale --replicas=0 statefulset <Release Name>-ac-rabbitmq -n <namespace>

  4. Scale down Rabbit MQ statefulset in the DR environment
    kubectl scale --replicas=0 statefulset <Release Name>-ac-rabbitmq -n <namespace>

  5. Initiate Failover for the storage account using below steps:

    Info

    >> These failover steps are not mandatory. These steps are mandatory only when the Azure File Share is not accessible due to any failure.

    >> As per Microsoft guideline, the time it takes to failover after initiation can vary though typically less than one hour.

    1. Login into your Azure Portal.

    2. Go to the Storage Account being used in the PVC of Prod environment.

    3. Within that storage account go to Data Management > Redundancy section.

    4. Click Prepare for failover and follow the instruction given on the screen. The failover process starts.

    5. Once the Account Failover process completes the data from secondary region will be accessible from the existing end point of the storage account.

  6. Copy web/repository folder from PROD to DR environment
    ./azcopy sync "https://[account].file.core.windows.net/[PVC]/web/repository/?[SAS]" "https://[account].file.core.windows.net/[PVC]/web/repository/?[SAS]" --recursive=true
    For Example:
    ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/web/repository/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/web/repository/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true

  7. Copy recovery folder from PROD to DR environment
    ./azcopy sync "https://[account].file.core.windows.net/[PVC]/recovery/?[SAS]" "https://[account].file.core.windows.net/[PVC]/recovery/?[SAS]" --recursive=true
    For Example:
    ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/recovery/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/recovery/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true

  8. Copy reprocessing folder from PROD to DR environment
    ./azcopy sync "https://[account].file.core.windows.net/[PVC]/reprocessing/?[SAS]" "https://[account].file.core.windows.net/[PVC]/reprocessing/?[SAS]" --recursive=true
    For Example:
    ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-e38eca50-xxxx-xxxx-xxxx-828efd6d49df/web/reprocessing/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-1a6f1c1b-xxxx-xxxx-xxxx-44ff485d707f/reprocessing/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true
  9. Copy Rabbitmq PVC content from PROD to DR environment
    ./azcopy sync "https://[account].file.core.windows.net/[PVC]/?[SAS]" "https://[account].file.core.windows.net/[PVC]/?[SAS]" --recursive=true --mirror-mode=true
    For Example:
    ./azcopy sync "https://fxxxxxxxxxxxxxxxxxxxxx1b6a.file.core.windows.net/pvc-ef345e9d-xxxx-xxxx-xxxx-54e38de9cbd1/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:18:24Z&st=2023-03-05T17:18:24Z&spr=https,http&sig=VxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxaXBmSAKm4%3D" "https://fxxxxxxxxxxxxxxxxxxx3aeb.file.core.windows.net/pvc-4a47fa22-xxxx-xxxx-xxxx-f6538dfff1a7/?sv=2021-06-08&ss=bfqt&srt=sco&sp=rwdlacupiytfx&se=2024-03-06T01:25:19Z&st=2023-03-05T17:25:19Z&spr=https,http&sig=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGazWA%3D" --recursive=true --mirror-mode=true
  10. Scale up RabbitMQ statefulset in the DR environment
    kubectl scale --replicas=1 statefulset <Release Name>-ac-rabbitmq -n <namespace>
  11. Scale up Listener in the DR environment
    kubectl scale --replicas=1 deployment <Release Name>-ac-runtime -n <namespace>
  12. Resume Scheduler in the DR environment
    https://<webapp_Gateway_URL>/event/schedulerservice?_dc=1670999141899&query=Resume&page=1&start=0&limit=25
  13. Update DNS record to route all request to DR environment

...

Tip
titleSwitching back to the Primary Database

Once the Primary Database is restored, you need to do the followings to switch back to it from the Sync Database:

  1. Ensure that the Primary Database is connected and performing Read/Write operations.
  2. Replicate the data from the Sync Database to the Primary Database.
  3. Update the DNS records to point to the Primary Database.

When the Storage is not accessible

Follow the steps below to point to secondary storage location:

  1. Login into your Azure Portal.

  2. Go to the Storage Account being used.

  3. Initiate “Account Failover”. The failover process starts.

  4. Once the Account Failover process completes the data from secondary region will be accessible from the existing end point of the storage account.

Appendix A: Getting Shared Access Signature (SAS) of your Azure Storage Account

  1. Login into your Azure Portal

  2. Go the Storage Account

  3. Within the Storage account go to Security + Networking > Shared Access Signature

    Image Added

  4. Enable the required permission as show below:
    Image Added

  5. Update the expiry date/time as per your need.

  6. In the Allowed Protocol enable HTTP and HTTPS both.

  7. Click Generate SAS and Connection String as shown below:
    Image Added

  8. Use the File Service SAS URL.