Current AC Cloud - DR Plan Test
- Subrat Sandilya
- Rajeev Rai
Requirement
To guarantee the AC4 cloud environments and their related resources are securely backed up and can be efficiently restored with minimal downtime in the event of any failure.
Scope of Backup & Restoration
The scope of restoration depends on the type of failure and its frequency. Velero tools are essential for backing up and restoring Kubernetes objects, playing a crucial role in establishing a dependable disaster recovery strategy.
Complete AKS cluster (Namespace with PVC) backup
Selected resources backup (e.g., PVC, Namespace)
AC4 cloud backend and log database backup
Restoration of selected resources
Restoration of AC4 cloud backend and log database
Policy Statements
All systems and databases must be preserved for AKS Cluster and applications to be completely recoverable, utilizing a combination of namespace duplicates, incremental backups, and database backups.
The volatility of data determines the frequency of backups; the retention period is determined by the criticality of data, with a minimum retention period of 7 days.
It is essential to securely store at least one fully recoverable version off-site, in a different Azure region, or off-site storage.
Backup documentation should clearly outline all critical data and necessary support items for recovery. Restoration documentation should encompass procedures for both individual system/application failures and total data center disaster scenarios.
It is crucial to regularly review and update backup and recovery documentation to align with new technology and business changes.
Annual testing of recovery procedures is imperative to ensure effectiveness.
Backup Points
Backup Component | Backup Method | Backup Performed By | Recovery Region | Backup/Retention Frequency | Retention Period |
Database | Azure Database | Azure | Geo Redundant | Point In Time | 7 Days |
Adeptia Connect Namespace | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment such as PVC, ConfigMap, and Certificates. | Adeptia (Velero Scheduled Job) | Depend on Cluster | Every 8 hours | 14 Days |
Ingress | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment (such as PVC, ConfigMap, Certificate, etc.). | Adeptia (Velero Scheduled Job) | Depend on Cluster | Every 8 hours | 14 Days |
Logs (EFK) Namespace | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment (such as PVC, ConfigMap, Certificate, etc.). | Adeptia (Velero Scheduled Job) | Depend on Cluster | Every 8 hours | 14 Days |
Monitoring (Grafana) Namespace | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment (such as PVC, ConfigMap, Certificate, etc.). | Adeptia (Velero Scheduled Job) | Depend on Cluster | Every 8 hours | 14 Days |
Restore\Recovery Points
Recovery Use Case | Recovery Method | Performed By | Recovery Region | Restore Duration |
Namespace is accidentally lost | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment (such as PVC, ConfigMap, Certificate, etc.). | Adeptia | The recovery region will remain the same if the existing cluster is running smoothly. If a new cluster needs to be set up in a different region, the recovery process will be carried out in that new region. | 2-4 hours |
PVC Failed | Using Velero backup PVC object will be resorted | Adeptia | Recovery will remain in the same region if the existing cluster is operational. If a new cluster needs to be set up in a different region, recovery operations will be conducted in that new region. | 2-4 hours |
Kubernetes Failed |
| Connectria/Azure |
| 1-2 days(As per Connectria) |
Database/Server Failed or accidentally lost | Restore the database from the database backup and verify the connectivity of the application. | Adeptia/Connectria |
| 2-4 hours |
Logs (EFK) Namespace | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment (such as PVC, ConfigMap, Certificate, etc.). | Adeptia | Recovery will remain in the same region if the existing cluster is operational. If a new cluster needs to be set up in a different region, recovery operations will be conducted in that new region. | 2-4 hours |
Monitoring (Grafana) Namespace | Using Velero backup, recover the entire namespace that includes components and objects of the application deployment (such as PVC, ConfigMap, Certificate, etc.). | Adeptia | Recovery will remain in the same region if the existing cluster is operational. If a new cluster needs to be set up in a different region, recovery operations will be conducted in that new region. | 2-4 hours |
Backup Location
Backups are stored in the following locations for restoration when needed:
Storage account:
accloudbackupswusAccount type: BlobStorage
Primary Location: West US, Secondary Location: East US
Backup Data storage in below containers or respective Customer
Backup Naming Convention
Backups from customer environments follow this naming convention:
daily<<customername>>prod_YYYYMMDDHHmmSS
Object\Namespace Included
Following the namespace and database configured part of the backup
AC4 Production
AC4 NonProduction
Production PVC
Non-Production PVC
Non Production
Production Database
Non-Production Database
Monitoring tool
EFK(log View) tool
Ingress Controller
Backup Log
The backup log includes details about all critical data, namespace, and necessary support items for recovery, ensuring the entire process is documented and reviewed regularly.
Database Backup Procedure
Azure provides automated backups for MySQL Flexible Server.
Backup retention can be configured up to 35 days (default is 7 days).
Diagram Explanation
The attached diagrams illustrate the backup and restore process for an application database in the Azure Cloud environment with Adeptia Connect 4.x. This setup ensures data integrity, business continuity, risk mitigation, and operational efficiency by enabling quick recovery from data-related incidents.
This document outlines a comprehensive backup and restore strategy for AC4 cloud environments, ensuring minimal downtime and efficient recovery in case of failures.
Storage Backup & Restore
The process demonstrates how Velero is utilized to manage backups and restore PVCs within an Azure cloud setting. This guarantees that data is securely backed up and can be easily restored in situations of data loss or migration requirements.
Sample Test Cases:
Use Case ID: UC01_Recover_Adeptia Objects
Use Case Scenario: Recover Adeptia Connect activities in Azure Kubernetes
Description: Details a scenario where a user has deleted some Adeptia activities and needs to recover them.
Pre-requisite: Access to the Azure portal and Kubernetes cluster is required.
TC ID | Test Case | Steps | Planned date | Execution Date | Executed By | Expected Result | Actual Result |
UC01_TC01 | Verify that the copy of below activities are created
| Log in to acdemo-sandbox.adeptia.com | mm/dd/yyyy | mm/dd/yyyy | John Smith | Copy of the activites are created | Pass |
UC01_TC02 | Verify if database backup is getting created successfully till the specified date. | Login to portal.azure.com with required credentials. | mm/dd/yyyy | mm/dd/yyyy | John Smith | Database backup should be created. | Pass |
UC01_TC03 | Verify that the activities are deleted. | Log in to acdemo-sandbox.adeptia.com | mm/dd/yyyy | mm/dd/yyyy | John Smith | Activities should be deleted. | Pass |
UC01_TC04 | Verify if the backup database is able to restore the deleted activities. | Change the database value in the yaml file | mm/dd/yyyy | mm/dd/yyyy | John Smith | The deleted activities should be now present | Pass |
UC01_TC05 | Verify that the restored activities data is consistent with the original data (e.g., timestamp, content). | Log in to acdemo-sandbox.adeptia.com | mm/dd/yyyy | mm/dd/yyyy | John Smith | The restored activities data is consistent with the original data. | Pass |
UC01_TC06 | Verify that Adeptia Connect is able to access and use the restored activities data correctly. | Log in to acdemo-sandbox.adeptia.com | mm/dd/yyyy | mm/dd/yyyy | John Smith | Adeptia Connect is able to access and use the restored activities data correctly. | Pass |
UC01_TC07 | Check for any errors or inconsistencies in the Adeptia Connect logs or database logs. | Log in to acdemo-sandbox.adeptia.com | mm/dd/yyyy | mm/dd/yyyy | John Smith | There are no errors or inconsistencies in the Adeptia Connect logs or database logs. | Pass |
UC01_TC08 | Perform any necessary configuration changes to the application to ensure that it is using the restored database instance. | Open any existing transaction. | mm/dd/yyyy | mm/dd/yyyy | John Smith | The application is able to function normally after the database recovery. | Pass |
UC01_TC09 | Create a new activity and cross check if it is getting saved in the new database correctly. | Create a new transaction from Adeptia Connect. | mm/dd/yyyy | mm/dd/yyyy | John Smith | Newly crated transaction and it's properties stored in the new database correctly. | Pass |
Conclusion:
The activities were successfully recovered in Adeptia Connect running in Azure Kubernetes through the database backup.
Use Case ID: UC02_Recover_Custom_Jar_or_PVC
Use Case Scenario: Recover a custom file or JAR from PVC or the whole PVC
Description: Describes a scenario where a user has deleted some custom files or JARs and wants to recover them.
Pre-requisite: A Kubernetes cluster is set up and running, with access to the Azure portal.
TC ID | Test Case | Steps | Planned Date | Execution Date | Executed By | Expected Result | Actual Result |
UC02_TC01 | Verify that the copy of the custom jar file is getting created. | Login to http://portal.azure.com with required credentials. | dd/mm/yyyy | dd/mm/yyyy | John Smith | Custom Jar copied successfully into share drive. | Pass |
UC02_TC02 | Verify that latest PVC backup is getting created successfully. | Take the latest backup of PVC through Velero backup command using command line. | dd/mm/yyyy | dd/mm/yyyy | John Smith | PVC backup should be created successfully. | Pass |
UC02_TC03 | Verify that the Namespace is getting deleted through command line. | Delete the namespace from the cluster using the command: | dd/mm/yyyy | dd/mm/yyyy | John Smith | Namespace should be deleted without any error | Pass |
UC02_TC04 | Verify that user is able to restore the Namespace PVC OR the jar file from Velero backup. | Restore the Namespace PVC through Velero restore command using command line | dd/mm/yyyy | dd/mm/yyyy | John Smith | The jar file and Namespace should be successfully recovered and accessible from its restored location. | Pass |
UC02_TC05 | Verify that if a user add a new jar then it is working fine with restored namespace PVC. | Add the new jar file(mapping functions) in shared drive and then login to application and check new methods or functions in data mapper. | dd/mm/yyyy | dd/mm/yyyy | John Smith | New mapping methods and functions should be added in datamapping activity with the restored Namespace PVC. | Pass |
UC02_TC06 | Verify that deleted namespace or jar file has been restored along with the PVC restore. | Check if the jar file OR Namespace has been restored by accessing it from the shared drive through command using the command line. | dd/mm/yyyy | dd/mm/yyyy | John Smith | The jar file should be accessible from the shared drive and located in the same directory where it was originally stored. | Pass |
UC02_TC07 | Verify that the custom functions are accesible through the Adeptia application once a jar file is recovered from storage. | Log in to acdemo-sandbox.adeptia.com | dd/mm/yyyy | dd/mm/yyyy | John Smith | Custom function should be accessible | Pass |
Conclusion:
The PVC and jar files was successfully recovered in Adeptia Connect running in Azure Kubernetes through the PVC backup.