Exporting objects using DevOps pipeline

You can use a CI/CD pipeline for exporting the Adeptia Connect objects from one environment to another automatically. The CI/CD pipeline uses the Export XML file that you must have created as a prerequisite for export operation, and generates an exported ZIP that contains the objects you want to migrate. After you create the Exported ZIP from the source environment, you need to import the objects to the new environment by running the import pipeline.

How the Export pipeline works

The diagram below represents how the export pipeline works, and automates the export process. 

When triggered, the pipeline performs the following sequence of actions.

• Connects to the GIT repository, pulls the Export XML file, and places it to a location in the PVC (shared folder).
• Pulls and deploys the migration utility helm chart to create the Exported ZIP using the Export XML file at a PVC (shared) location.
• Downloads the newly created Exported ZIP, and pushes it to the GitHub repository. 
• Deletes the migration workspace created during the export process.

To start with exporting the objects, you first need to create an export pipeline having all the required parameters for exporting the objects.

After you create the pipeline, you need to trigger the pipeline to perform the export operation.

Creating and triggering the Export pipeline

To create the pipeline in Jenkins using the export pipeline provided by Adeptia, follow the steps given below.

1.  Log in to the Jenkins with admin privileges.
   

2.  Select New Item.
   

   
   

3. Enter a name for the Export pipeline, and then select Pipeline.

   

4. Click OK.

5. Copy the content from the provided export pipeline file.

    Export Pipeline

   import jenkins.model.Jenkins
   import java.nio.file.Path;
   import java.nio.file.Paths;
   import java.io.File;
   
   
   /*
       This pipeline used to deploys migration promotion through operations like export from one build environment to another.
   
       Pipeline is made up of following steps
       1. Init Perameters
       2. Pull export XML file from GitHub
       3. Upload XML to k8 shared PVC
       4. Download Helm chart & deploy migration (export)
       5. Download solution zip from k8 shared PVC
       6. Push soution zip to GitHub
   	7. Clean up workspace
   
   	Pre-requisite
   	a) Tools/Plugins needs to install:
   		1. Helm
   		2. kubectl client
   		3. Jenkins
   		4. Java 1.8+
   	b) OS Linux
   	c) Jenkins plugins
   		1. Kubernetes
   		2. Git (git plugin 4.8.3, Git client plugin 3.9.0)
   		3. Mask Password
   		4. Credentials Binding Plugin (1.27)
   		5. Parameter Separator
   		6. BlueOcean (Optional)
   		
   	Usage:
   		Steps to create pipeline using jenkinsfile.
   		1. Login into the Jenkins GUI with admin privileges.
   		2. create a pipeline by choosing New Item > Pipeline.
   		3. Copy/past containt of jenkinsfile to Pipeline Definition area.
   		4. Uncheck checkbox "Use Groovy Sandbox".
   		5. Save the pipeline.
   		6. Trigger the pipeline once to initilaize parameters.
   	
   */
   
   /*
       Upload file to Kubernetes PVC
    */
   def uploadToSharedPVC (NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, SERVER_URL, SRC_FILE_PATH, TRG_FILE_PATH) {
       echo "Upload files to K8 shared PVC"
   		withKubeConfig([credentialsId: K8_CREDENTIALS_ID, serverUrl: SERVER_URL]) {
   					try {
   						  sh '''#!/bin/bash
   							kubectl config use-context '''+CLUSTER_CONTEXT+'''
   							TRG_FILE_PATH='''+TRG_FILE_PATH+'''
   							if [[ ${TRG_FILE_PATH::1} == "/" ]]
   								then
   								  TRG_FILE_PATH=${TRG_FILE_PATH:1};
   								else
   								  echo "Forward shash(/) already removed "; fi
   							podname=$(kubectl -n '''+NAMESPACE+''' get pods | grep -m 1 autoscaler | awk '{print $1}')
   							kubectl -n '''+NAMESPACE+''' cp '''+SRC_FILE_PATH+''' ${podname}:${TRG_FILE_PATH}
   							jobname=$(kubectl -n '''+NAMESPACE+''' get jobs | grep -m 1 migration | awk '{print $1}')
   							if [[ -n "$jobname" ]]; then
   								kubectl -n '''+NAMESPACE+''' delete job ${jobname}
   								else 
   									echo "Migration resource does not exist"
   							fi
   											'''				
   						  
   						} catch (err) {
   								echo "Caught: ${err}. Error in uploading file."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   						}
   }
   
   /*
       Download file from Kubernetes PVC
    */
   def downloadFromSharedPVC (NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, SERVER_URL, SRC_FILE_PATH, TRG_FILE_PATH) {
       echo "Download files from K8 shared PVC"
   		withKubeConfig([credentialsId: K8_CREDENTIALS_ID, serverUrl: SERVER_URL]) {
   					try {
   						sh '''#!/bin/bash
   							SRC_FILE_PATH='''+SRC_FILE_PATH+'''
   							if [[ ${SRC_FILE_PATH::1} == "/" ]]
   								then
   								  SRC_FILE_PATH=${SRC_FILE_PATH:1};
   								else
   								  echo "Forward shash(/) already removed "; fi
   							kubectl config use-context '''+CLUSTER_CONTEXT+'''
   							podname=$(kubectl -n '''+NAMESPACE+''' get pods | grep -m 1 autoscaler | awk '{print $1}')
   							kubectl -n '''+NAMESPACE+''' cp ${podname}:${SRC_FILE_PATH} '''+TRG_FILE_PATH+'''
   								'''											
   						} catch (err) {
   								echo "Caught: ${err}. Error in downloading file from K8 PVC."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   						}
   }
   
   /*
       Pull Helm Chart
    */
   def pullHelmChart (HELM_REPO_URL, CHART_NAME) {
       echo "Pull Helm Chart ("+CHART_NAME+") from Artifact Hub"
   					try {
   						sh '''#!/bin/bash
   							helm repo add adeptia-connect-migration '''+HELM_REPO_URL+'''
   							helm pull adeptia-connect-migration/'''+CHART_NAME+''' --untar					
   							'''
   						} catch (err) {
   								echo "Caught: ${err}. Error in pulling Helm chart from repo."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   		
   }
   
   /*
       Deploy Helm to Kubernetes cluster
    */
   def deployToCluster (NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, DATABASE_CREDENTIALS_ID, SERVER_URL, EXPORT_ZIP_PATH, MIGRATION_XML_FILE_PATH) {
       echo "Deploy Helm chart to Kubernetes cluster"
   					try {
   						def BACKEND_DB_USERNAME = getUserName(DATABASE_CREDENTIALS_ID);
   						def BACKEND_DB_PASSWORD = getPassword(DATABASE_CREDENTIALS_ID);
   						withKubeConfig([credentialsId: K8_CREDENTIALS_ID, serverUrl: SERVER_URL]) {
   						//hide password field
   						wrap([$class: 'MaskPasswordsBuildWrapper', varPasswordPairs: [[password:BACKEND_DB_PASSWORD], [password:BACKEND_DB_USERNAME]]]) { 
   						  sh '''#!/bin/bash
   							kubectl config use-context '''+CLUSTER_CONTEXT+'''
   							helm upgrade -i migration migration -f migration/values.yaml --set environmentVariables.BACKEND_DB_URL=${BACKEND_DB_URL} --set environmentVariables.BACKEND_DB_USERNAME='''+BACKEND_DB_USERNAME+''' --set environmentVariables.BACKEND_DB_PASSWORD='''+BACKEND_DB_PASSWORD+''' --set environmentVariables.BACKEND_DB_DRIVER_CLASS=${BACKEND_DB_DRIVER_CLASS} --set environmentVariables.BACKEND_DB_TYPE=${BACKEND_DB_TYPE} --set environmentVariables.EXPORT_ZIP_PATH='''+EXPORT_ZIP_PATH+''' --set environmentVariables.MIGRATION_XML_FILE_PATH='''+MIGRATION_XML_FILE_PATH+''' --set environmentVariables.LOG_IDENTIFIER=${LOG_IDENTIFIER} --set environmentVariables.OPERATION=${OPERATION} -n '''+NAMESPACE+'''
   											'''	
   							}
   						}
   						} catch (err) {
   								echo "Caught: ${err}. Error in deploying Helm chart."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   }
   
   /*
       Wait until deployment finish on Kubernetes cluster
    */
   def waitUntilDepoymentComplete(NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, SERVER_URL, POD, time_out) {
       echo "Fetching pod status"
   					try {
   						int inter = 5, count = 1;					
   						withKubeConfig([credentialsId: K8_CREDENTIALS_ID, serverUrl: SERVER_URL]) {
   						sh('kubectl config use-context ${CLUSTER_CONTEXT};')
   						while (true) {
   								def status = sh script: "kubectl -n ${NAMESPACE} get pods | grep -m 1 ${POD} | awk '{print \$3}' ", returnStdout: true
                                   if (status.toString().trim().contains("Completed")) {
                                       break;
                                       }
   								else if (status.toString().trim().contains("Error")) {
   									error("Caught: Migration deployment failed due to error. Please check migration logs.")
   									currentBuild.result = 'FAILURE'
                                       break;
                                       }
                                  sleep(inter)
   							   echo count+" retry in "+inter*count+" seconds."
   							   count++
   							   if ((count)>=((time_out-10)/inter)) {
   									error("Caught: Migration deployment taking more then ideal time. Please check migration logs.")
   									currentBuild.result = 'FAILURE'
   									break;
   							   }
                               }
   						}	
   						} catch (err) {
   								echo "Caught: ${err}. Error in fetching pod status. Migration deployment taking more then ideal time. Please check migration logs."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   }
   
   /*
       Push soution Zip to GitHub repository
    */
   def pushToGitHub (GIT_BRANCH, GIT_CREDENTIALS_ID, GIT_REPO_URL, FILE_PATH) {
       echo "Pushing file ("+FILE_PATH+") to GitHub repo"
   				withCredentials([gitUsernamePassword(credentialsId: GIT_CREDENTIALS_ID, gitToolName: 'git-tool')]) {
   					try {
   						def gitUser = getUserName(GIT_CREDENTIALS_ID);
   							sh('sleep 10')
   							sh('git config --global user.name "'+gitUser+'"')
   							sh('git config --global user.email "you@example.com"')
   							sh('git add '+FILE_PATH)
   							sh('git commit -m "auto commit message" ')
   							sh('git push ${GIT_REPO_URL} HEAD:'+GIT_BRANCH)
   						} catch (err) {
   								echo "Caught: ${err}. Error in pushing file to Github."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   					}
   }
   
   /*
       Generate rollback soution Zip file path
    */
   def convertRollbackZipPath(FILE_PATH) {
   					def rollbackZipPath = null
   					def Append = "Rollback_"
   					try {
   						Path path = Paths.get(FILE_PATH);
   						def fileName=path.getFileName().toString()
   						def parentDir=path.getParent().toString()
   						rollbackZipPath=parentDir + File.separator + Append + fileName
   						if(isUnix()){
   						rollbackZipPath=rollbackZipPath.replace("\\", "/")
   						}
   						} catch (err) {
   								echo "Caught: ${err}. Error in generating rollback soution Zip file path."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   					return rollbackZipPath
   }
   
   /*
       Get username from credentials id
    */
   def getUserName(id) {
   	def userName = null
   	withCredentials([usernamePassword(credentialsId: id, passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
   					try {
   						userName = USERNAME
   						} catch (err) {
   								echo "Caught: ${err}. Error in extracting username from "+id+" ."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   	}
   	return userName
   }
   
   /*
       Get password from credentials id
    */
   def getPassword(id) {
   	def password = null
   	withCredentials([usernamePassword(credentialsId: id, passwordVariable: 'PASSWORD', usernameVariable: 'USERNAME')]) {
   					try {
   						password = PASSWORD;
   						} catch (err) {
   								echo "Caught: ${err}. Error in extracting password from "+id+" ."
   								error("Caught: ${err}")
   								currentBuild.result = 'FAILURE'
   							}
   	}
   	return password
   }				  
   					
   pipeline {
   	
   	// Global default variables
       environment {
   		//manage deployment status timeout
   		time_out = 300
       }
   	parameters{
   		//separator(name: 'separator-ce1a9ef5-cd10-4002-a43f-8ae24d9d0bb3', sectionHeader: '''Helm Chart Parameters''', sectionHeaderStyle: 'background-color:#eeeee4;font-size:15px;font-weight:normal;text-transform:uppercase;border-color:gray;', separatorStyle: '''font-weight:bold;line-height:1.5em;font-size:1.5em;''') 
   		string(defaultValue: '', description: 'ArtifactHub Helm chart URL e.g. https://adeptia.github.io/adeptia-connect-migration/charts', name: 'HELM_REPO_URL', trim: true) 
   		string(defaultValue: '', description: 'Name of Helm chart to be downloaded from ArtifactHub repository e.g. migration', name: 'CHART_NAME', trim: true)
   		
   		//separator(name: 'separator-ce1a9ef5-cd10-4002-a43f-8ae24d9d0bb3', sectionHeader: '''GitHub Parameters''', sectionHeaderStyle: 'background-color:#eeeee4;font-size:15px;font-weight:normal;text-transform:uppercase;border-color:gray;', separatorStyle: '''font-weight:bold;line-height:1.5em;font-size:1.5em;''')
   		string(defaultValue: '', description: 'GitHub credentials ID configured in Jenkins e.g. gitCredential_id', name: 'GIT_CREDENTIALS_ID', trim: true) 
   		string(defaultValue: '', description: 'GitHub server URL e.g https://github.com/adeptia/migration-defination.git', name: 'GIT_REPO_URL', trim: true)
   		string(defaultValue: '', description: 'GitHub Branch name e.g. main', name: 'GIT_BRANCH', trim: true)
   		string(defaultValue: '', description: 'Path to upload zip file to GitHub. e.g. test/SA_PF.zip', name: 'GIT_EXPORT_ZIP_PATH', trim: true)
   		string(defaultValue: '', description: 'export xml file path to download from GitHub. e.g. test/export.xml', name: 'GIT_EXPORT_XML_PATH', trim: true)
   
   		//separator(name: 'separator-ce1a9ef5-cd10-4002-a43f-8ae24d9d0bb3', sectionHeader: '''Migration Parameters''', sectionHeaderStyle: 'background-color:#eeeee4;font-size:15px;font-weight:normal;text-transform:uppercase;border-color:gray;', separatorStyle: '''font-weight:bold;line-height:1.5em;font-size:1.5em;''')
   		string(defaultValue: '', description: 'Location of export xml file. eg. export', name: 'OPERATION', trim: true)
   		string(defaultValue: '', description: 'Migration export zip path. e.g. /shared/SA_PF.zip', name: 'EXPORT_ZIP_PATH', trim: true)
   		string(defaultValue: '', description: 'Migration export xml file path. e.g. /shared/export.xml', name: 'MIGRATION_XML_FILE_PATH', trim: true)
   		string(defaultValue: '', description: 'Migration log identifier to capture logs from MS environment.', name: 'LOG_IDENTIFIER', trim: true)
   		
   		//separator(name: 'separator-ce1a9ef5-cd10-4002-a43f-8ae24d9d0bb3', sectionHeader: '''K8 Cluster Parameters''', sectionHeaderStyle: 'background-color:#eeeee4;font-size:15px;font-weight:normal;text-transform:uppercase;border-color:gray;', separatorStyle: '''font-weight:bold;line-height:1.5em;font-size:1.5em;''')
   		string(defaultValue: '', description: 'Credentials ID configured in Jenkins to access K8 cluster e.g k8credentials', name: 'K8_CREDENTIALS_ID', trim: true)
   		string(defaultValue: '', description: 'URL to access K8 cluster e.g. https://*******-dns-2ce021bb.hcp.eastus.azmk8s.io:443. You can get the server Url from K8 config file.', name: 'SERVER_URL', trim: true)
   		string(defaultValue: '', description: 'Cluster context to access K8 cluster e.g. adeptia-context', name: 'CLUSTER_CONTEXT', trim: true)
   		string(defaultValue: '', description: 'K8 cluster name space deployment where Connect microservices deployed e.g. adeptia', name: 'NAMESPACE', trim: true)
   		string(defaultValue: '', description: 'URL of database backend bind with application.', name: 'BACKEND_DB_URL', trim: true)
   		string(defaultValue: '', description: 'Credentials ID configured in Jenkins to access database.', name: 'DATABASE_CREDENTIALS_ID', trim: true) 
   		string(defaultValue: '', description: 'Driver class of database e.g com.microsoft.sqlserver.jdbc.SQLServerDriver', name: 'BACKEND_DB_DRIVER_CLASS', trim: true)
   		string(defaultValue: '', description: 'Database type e.g SQL-Server.', name: 'BACKEND_DB_TYPE', trim: true)
   		
   		}
   	
       /*
   		agent {
           	label 'LinuxAgent' 	
       }
   	*/
   	agent any
   
   stages {
           stage('Pull XML from GitHub)') {
   			steps {
   				echo 'Checkout from GitHub'
   					checkout([$class: 'GitSCM', branches: [[name: '*/'+GIT_BRANCH]], extensions: [], userRemoteConfigs: [[credentialsId: GIT_CREDENTIALS_ID, url: GIT_REPO_URL]]])					 
   			}
           }
   		stage('Upload files to PVC') {
            	steps {
   				echo 'Uploading export xml file'
   					uploadToSharedPVC (NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, SERVER_URL, GIT_EXPORT_XML_PATH, MIGRATION_XML_FILE_PATH)
            	}
           }
   		stage('Pull Helm chart & Deploy Migration') {
   			steps {
   				echo 'Pulling Helm Chart'
   					pullHelmChart (HELM_REPO_URL, CHART_NAME)
   				
   				echo 'Deploying Helm Chart'
   					deployToCluster (NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, DATABASE_CREDENTIALS_ID, SERVER_URL, EXPORT_ZIP_PATH, MIGRATION_XML_FILE_PATH)
   					timeout(time: env.time_out, unit: "SECONDS"){
   					waitUntilDepoymentComplete(NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, SERVER_URL, 'migration-', env.time_out.toInteger())
   					}
   			}
   		}
   		stage('Download Zip from PVC') {
            	steps {
               	downloadFromSharedPVC (NAMESPACE, CLUSTER_CONTEXT, K8_CREDENTIALS_ID, SERVER_URL, EXPORT_ZIP_PATH, GIT_EXPORT_ZIP_PATH)
            	}
           }
   		stage('Push Zip to GitHub') {
   			steps {						
   				pushToGitHub (GIT_BRANCH, GIT_CREDENTIALS_ID, GIT_REPO_URL, GIT_EXPORT_ZIP_PATH)	
   			}
           }
   	}
   	post('Clean-up') {
   			always {
   				echo 'Cleanup workspace'
   				cleanWs()
   			}
   			success {
   				echo 'Pipeline succeeded!'
   			}
   			unstable {
   				echo 'Pipeline unstable :/'
   			}
   			failure {
   				echo 'Pipeline failed :('
   			}
   		}
   }

6. In the Pipeline Definition section, paste the copied content and uncheck the Use Groovy Sandbox checkbox. 

   

   Important

   If you are using Jenkins on Windows OS, and have created an agent on Linux OS, you need to do the followings in the export pipeline file.

   1. Uncomment the following code snippet.

   /*
   		agent {
           	label 'LinuxAgent' 	
       }
   	*/

   Where,

   LinuxAgent is the name of the agent that you have created.

   2. Comment the following lines of code.

   agent any

7. Click Save.

8. On the screen that follows, click Build Now.
   
   As you build the pipeline for the very first time, all the parameters get initialized. 

9. Refresh the page.
   The Build Now option now changes to Build with Parameters.

10. Click Build with Parameters.

    

    You will see all the parameters inherited from the export pipeline file. 

11. Enter the parameter values as per your requirement.

    It is mandatory to provide a valid value for all the Jenkins parameters.
     Click here to expand the list of Export parameters

    Parameters	Value	Description
    Helm Chart
    HELM_REPO_URL	https://adeptia.github.io/adeptia-connect-migration/charts	Migration Helm chart repository URL.
    CHART_NAME	migration	Migration Helm chart name.
    GitHub
    GIT_CREDENTIALS_ID	<credential ID generated by Jenkins>	Credential ID for GitHub in Jenkins. Refer to the prerequisites for more details.
    GIT_REPO_URL	https://github.com/adeptia/migration-defination.git	URL of the GitHub repository.
    GIT_BRANCH	main	GitHub branch name.
    GIT_EXPORT_XML_PATH	xml/export.xml	Path of the export XML in the GitHub repository.
    GIT_EXPORT_ZIP_PATH	test/SA_PF.zip	Path of the exported ZIP in the GitHub repository.
    Migration
    OPERATION	export	The type of operation for which deployment will be performed.
    MIGRATION_XML_FILE_PATH	/shared/migrationtest1/export.xml	Path of the export XML in the PVC.
    EXPORT_ZIP_PATH	/shared/migrationtest1/SA_PF.zip	Path of the exported ZIP in the PVC.
    LOG_IDENTIFIER	Test_Identifier_Tag	Log identifier to capture logs from MS environment.
    Kubernetes Cluster
    K8_CREDENTIALS_ID	<credential ID generated by Jenkins>	Credential ID for Kubernetes in Jenkins. Refer to the prerequisites for more details.
    SERVER_URL	https://<host name of the Kubernetes cluster>:<Port number>	URL of the Kubernetes cluster.
    CLUSTER_CONTEXT	test-dev	Kubernetes cluster context.
    NAMESPACE	namespace	Kubernetes cluster namespace for deployment.
    Database
    BACKEND_DB_URL	jdbc:sqlserver://<DB Hostname>:<Port number>;database=<Backend Database Name>	Backend database URL used in the deployment of migration Helm chart. Database info configured in value.yaml file.
    DATABASE_CREDENTIALS_ID	<credential ID generated by Jenkins>	Credential ID for database in Jenkins. Refer to the prerequisites for more details.
    BACKEND_DB_DRIVER_CLASS	com.microsoft.sqlserver.jdbc.SQLServerDriver	Backend database driver class used in the deployment of migration Helm chart. Database info configured in values.yaml file.
    BACKEND_DB_TYPE	SQL-Server	Backend database type used in the deployment of migration Helm chart. Database info configured in values.yaml file.

12. Click Build to trigger the pipeline.

    • The exported ZIP is created and pushed to the GitHub repository. 

    • All the log statements generated by the migration utility during the export process are tagged with a unique identifier. This allows you to search for this unique identifier in the centralized logging system, and fetch all the associated logs.