Let’s say you’re on a team inside a large organization. Your team works on multiple services, each of which has external dependencies on services from other teams. Anytime you want to deploy to an environment, it’s critical that these external dependencies are already up and running or are deployed along with your service.
Many platforms support manifest-based deployments where a list of services can be deployed together to a certain namespace. But what if your application components exist across multiple platforms — what if the deployment methods are different for each service? And what do you do when the service APIs you depend on also have their own dependencies?
Companies often struggle with deploying their applications. The larger the applications, the more difficult it can be to set them up inside an environment. With large enterprises, having a web of services that exists across multiple platforms is all too common. Typically, teams try to address this challenge by gathering their list of service dependencies and manually kicking off deployments for each of them. However keeping track of each dependency and each nested dependency along with the specific version numbers needed can lead to massive application deployment overhead. The more manual steps in the deployment process, the more error-prone that process becomes.
That’s where orchestrating manifest-based deployments with Jenkins comes in.
Simply put, deployments should be automated.
While the best solution would be to re-architect your application into microservices that can act more autonomously, that process can be long and labor intensive. We sought a more immediate solution. Instead of a team having to deploy each piece manually, we recently came up with a solution where the team lists all their dependencies and then uses Jenkins automation to handle the deployments.
So how do we orchestrate this process starting from scratch? Since things can get tricky when trying to handle nested dependencies, we’ll only worry about immediate dependencies for now. First thing’s first — we know we’ll need a manifest, or a list of services that get deployed. So our starting point can be a Jenkins pipeline that takes in one parameter: a manifest file.
Since the methods can vary depending on the platform, the manifest will need to tell us not only what to deploy but also how to deploy it. This manifest will also need to be in a format that Jenkins can read and easily parse.
When orchestrating manifest-based deployments with Jenkins, the two file formats that are easiest for Jenkins to read are JSON and YAML. Looking at what’s available through Jenkins, all the logic for orchestrating the deployments can exist inside Groovy scripts and can be standardized using Groovy shared libraries. While Groovy already has libraries for parsing both JSON and YAML, thankfully Jenkins also has a Pipeline Utility Steps plugin, which eases the process. In the examples below, I’ll use JSON.
The manifest can look like this to start:
We can easily read in this file like so:
At this point, our manifest exists as a Groovy object, and we can easily pull values out to start reading. We will need to iterate the service list and call the appropriate methods for deploying. In Groovy, this iteration can look like this:
In the example above, we’re assuming that we have shared library methods for deploying to different platforms. Deployments can be handled by reading from the JSON manifest what platform we would like to deploy to and then calling the appropriate method where we simply pass in the name and version of the service.
Another idea that we had was to set up jobs inside Jenkins that strictly do deployments. After setting up these jobs, we can call those jobs and pass in any necessary parameters.
A little more configuration is available when using the Jenkins jobs instead of just calling the shared library methods, but ultimately it isn’t too important which method you choose. What is important is that the deployment logic is abstracted away from the orchestration process, which makes the orchestration code much less bloated and ensures the deployment methods have many more opportunities for reuse.
In discussing orchestrating manifest-based deployments with Jenkins, so far I’ve only been talking about deploying immediate dependencies. But what if we want to try to deploy nested dependencies as well? Here’s where things can get tricky.
To know what our nested dependencies are, each of the immediate dependencies needs to have its own manifest files that we can retrieve. These manifests need to be collected, and the list of components to be deployed needs to be coalesced into a larger list. We also need a little more information passed into our original manifest, specifically the location where the manifest lives.
The manifest location can be anything from a git URL to an Artifactory endpoint. Once we know where all the manifests are located, we can collect them and combine the dependency lists.
First, we pull down the manifest into our workspace using wget. Next, we use the Pipeline Utility Steps plugin like before and read in the JSON file. From here, Groovy makes it simple to combine the lists and then filter out any duplicate dependencies that might have been shared across components.
Benefit of Orchestrating Manifest-Based Deployments with Jenkins
The code I’ve shared here should provide a good starting point for orchestrating manifest-based deployments with Jenkins. By taking care of dependencies through automation on the front end, you’ll have less scrambling to track down missing dependencies and manually deploy those dependencies on the back end.