Oftentimes issues in technology have a funny way of repeating themselves. We see large corporations and engineering teams fall into the same problem: low frequency deployments and being slow to roll out new features. By taking the time to implement a proper GitOps strategy, not only will you help increase the frequency of your deployments, you will also experience many other key DevOps benefits. This blog aims to educate readers on a trunk-based GitOps approach for IaC deployment repos and the benefits that this structure will bring.
What is GitOps?
GitOps is a framework that takes best practices from DevOps patterns such as version control, declarative code, collaboration, continuous integration (CI) and continuous deployment (CD) patterns, and then applies them to your infrastructure and deployment automation. OpenGitOps defines the four crucial GitOps principles below:
- Declarative
A GitOps-managed system must have state defined and expressed declaratively. - Versioned and Immutable
Desired state must be stored in a way that forces immutability, versioning, and retains a complete version history. - Pulled Automatically
Software agents must be available to pull desired state declarations from the source in an automated manner. - Continuously Reconciled
Software agents continuously observe the actual system state and apply new changes for the latest desired state.
Why Should You Care About GitOps?
As discussed above, there is a very common issue of handling deployments and infrastructure manually. Now that many software development life cycles have been automated through DevOps practices, there is even more of a demand and need for automated deployments and infrastructure automation. Organizations that have matured their DevOps practices and implemented a GitOps strategy are able to deploy to production environments much more quickly and with less risk. Listed below, but not limited to, are the many benefits that individuals, teams, and organizations can experience with a GitOps implementation:
- Increased Developer Productivity
Developers no longer need to worry about time consuming manual deployments and scheduled deployment times. - Version Controlled Systems
Version control allows teams to easily determine what is deployed at any given time to their environments, and allows for easy rollbacks when changes fail. - Improved Software and Infrastructure Stability
With CI and CD both implemented, there is less room for manual error at any stage of the software development lifecycle. Automated pipeline checks also ensure high quality code is enforced and new bugs are not introduced during the deployment process. - Higher Reliability
With a declarative defined state, you can understand and limit the risk of what is deployed in your environments. - Consistency and Standardization Across Environments
Environments no longer include manual and unknown changes; instead everything is defined in source and can be compared.
Infrastructure As Code with GitOps
GitOps can encompass a variety of things; for this blog we will simply look at Liatrio’s guidance for your infrastructure strategy. Our stance using Trunk-Based Gitops relies greatly on the combination of HashiCorp's Terraform and Gruntwork’s Terragrunt. However, the benefits of a DevOps solution is the ability to substitute tools in and out for your desired needs, the principle and structure behind the trunk-based style should still remain the same.
- Terraform (using HCL) declaratively defines what things should exist, their properties, and their relationships.
- Terragrunt orchestrates Terraform; codifying how to actually apply one or more sources of Terraform, normalizing ways to provide contextual values, and general quality of life improvements over directly-handled Terraform.
Key GitOps Concepts
In order to be the most effective, Liatrio aligns with the following concepts:
Terraform source modules should live independently in their own git repositories
This allows for modules to be managed and versioned independently from the source of truth for your environments. These shared repositories allowed for product and operation teams to contribute changes and constantly allow for changes to the modules without affecting your deployed environment.
Modules should be versioned and tagged through automation
Consistency and reliability will be enforced through the use of continuous automation to ensure your modules are in deployable states. Additionally, a version control system like semantic versioning should be applied to help identify which type of changes are being made to the module and the severity of that change.
A separate Terragrunt Git Repository should be created for your Continuous Delivery
This module should represent the state of your environment and should only contain remote references to your source modules through the use of semver tags. Environment specific values should be defined in value files to be passed into the modules being consumed.
Merge Request must be leveraged
By utilizing merge requests, you can put in place protection for your environments. This will force manual approvals for higher level environments and prevent accidental changes being merged at any given time or prevent users from bypassing higher level environment requirements.
Repo State should always match environment state
By maintaining a matching repository state to your environment state, you could easily see what is deployed at any given time and make changes as necessary. One good example of this is rollbacks to modules that introduce unintended bugs.
Trunk-Based GitOps
Trunk-based or directory-based GitOps is a method that defines all the environments on a single main branch. Each environment will have a separate directory, and inside each environment directory, will be defined values and versions for that specific environment. Upon merged changes through the use of merge requests, a pipeline would detect new changes and apply those changes directly to the affected environments.
Common Issues with GitOps Strategies
Many GitOps strategies have similar problems, and by using a trunk-based approach you could solve most of these problems. Listed below are common issues we see with alternative patterns and how trunk-based aims to solve them.
Duplication of Code & Changes
One big problem with a strategy such as a branch-based approach is the duplication of code or changes that are required to make a single change. Trunk-based promotes “DRY” infrastructure changes on a single branch where most code definitions could be shared, and changes to environments only need to be done in one place.
Environment Drift
Another big problem we notice when defining GitOps strategies in a branch-based model is drift amongst environments. This is typically due to discrepancies between higher level environments and mostly happens when lower environments are changing much more frequently than higher level environments. Trunk-based Gitops allows for changes against all environments in a single change request to help reduce this drift.
Maintenance of Multiple Sources of Truth
Branch-based GitOps for your infrastructure requires you to maintain multiple branches at all times. This ultimately leads to drift and confusion of what is your source of truth. With trunk-based, only a single branch acts as the source of truth and only one branch is ever maintained.
Rolling back is challenging
When using multiple branches to define your environments, a user must make multiple and duplicated change requests to roll back. This introduces more downtime and bigger risks since changes are not easily able to be rolled back. With a trunk-based approach, a single change could consecutively apply to all your environments and quickly rollback to fix an unintended bug or vulnerability.
On top of all these problems a trunk-based strategy solves, it is also promotes other benefits such as:
Single source of truth
All changes are defined on a single branch and this branch serves as a single source of truth. At any given moment a user should be able to identify what is deployed in any given environment based on the trunk of the branch.
Environments could easily be replicated
Since all the changes are stored on a single branch and values are simply passed in through config files, a new directory structure could be replicated or cloned to spin up new environments. This allows for multiple development teams to have their own environments for testing.
Conclusion
Every problem has multiple solutions, but here at Liatrio, we believe a trunk-based Gitops approach to your infrastructure is your best solution. Even though Terraform and Terragrunt played big roles in our solution, DevOps allows for the flexibility to easily incorporate different tools but still follow the same GitOps pattern. Hopefully this blog was informative enough to help you understand the importance of trunk-based Gitops, be sure to reach out for any additional help with setting up your repositories.
