While Hashicorp’s Terraform is one of the most useful tools available for creating and managing infrastructure, it has some limitations. Terraform’s official providers usually work great for large and mainstream tools, such as AWS, Helm, and Vault, but if you need support for less-popular tools, you’ll have to rely on community providers. Of course, these providers may not be available, and the ones that are available aren’t guaranteed to support your use case.
We found ourselves in this situation when using Harbor, a Cloud Native Computing Foundation (CNCF) artifact registry. Community providers for Harbor did exist, but they either didn’t support the version of Harbor we were running or didn’t have adequate test coverage. In response, we decided to write our own Terraform Provider for Harbor.
Let’s take a look at the mechanics of Terraform and walk through how we extended its functionality by creating our own provider plugin.
Writing the Harbor Client
The first thing our new Harbor provider needed was the ability to interact with the Harbor API. Because no official Golang client was available, we had to write our own. We kept the client code separate from any Terraform-related code to enable client code reuse.
In our Harbor provider, the “harbor” package contains our client code. At the most basic level, the package contains:
- Entry point, which initializes a Harbor client and handles authentication
- Series of files corresponding to API resources, each of which defines a model used by the Harbor API and handles the serializing and deserializing of model data
- Utility functions, such as helpful error-handling functions and helper functions for acceptance tests
If you’re interested in writing your own API client but lack experience, follow these simple tips:
- Do not hard code API endpoints throughout the code. Any path that needs to be defined can be set a single time and referenced whenever it’s needed. That’ll save a lot of headaches if the API changes at a later time.
- Make sure your client models all of the data that comes from the API. For example, the Harbor API can provide information about the creation time of certain resources. The client package preserves this information even though Terraform doesn’t use it. Preservation of this information makes the client available for general purpose use beyond its use for the Terraform plugin.
- Use the client for all communications with the API. It’s easy to make a quick, one-off call to an API endpoint directly from your provider, but taking the time to add the functionality to your client helps keep the code more consistent. You’ll thank yourself if you need to reuse the API call later.
Writing the Provider
Once we could interact with the Harbor API, we began writing the provider itself. In the Harbor provider, the “provider” package contains our provider code. All Terraform provider plugins make heavy use of terraform-plugin-sdk, which provides the tools that the plugin uses to interact with Terraform.
When writing the provider plugin, we had to handle information specific to the provider itself. For example, a provider needs to define what infrastructure and configuration it can manage, as well as how to manage those details. However, we didn’t have to handle things like parsing HCL, reading and writing state, or deciding how to order configuration changes. Terraform Core handled those details for us.
To create our provider, we needed to define schemas to tell Terraform what data the provider requires. If you’ve used any official Terraform providers, you know that they come with three fundamental types:
- Providers - define how a provider should be initialized and how it should authenticate
- Data - represent data that can be pulled from an API without necessarily creating any infrastructure
- Resources - denote actual infrastructure or configuration changes that will be made to the service being managed by Terraform
The first step in writing the Harbor provider was to define the provider schema. We denoted the information the provider needs to initialize, in this case the URL of the instance to manage, along with a username and password for authentication. In addition, we defined the resources that the provider manages. In our case, it supports management of Harbor projects and robot accounts but doesn’t support any data sources. The provider plugin is fairly simple, but the benefit of writing our own provider was that we could make it as basic or as complex as necessary.
Next, we defined each resource the provider supports by defining specific fields required for the resources, such as a “name” field for Harbor projects. Field definitions can also include fields provided by the API as output rather than provided by the user as input. With Harbor, one example is a robot account token, which Harbor generates when a new robot account is created and which can’t be specified by a user.
In addition to defining the schema, each type must tell Terraform how it is managed. For our Harbor provider type, we initialized our Harbor client and passed that client to later functions. For our resources, we created functions specifying what Terraform should do when a resource is read, created, updated, or deleted. At their most basic level, these functions simply pass data from Terraform to the Harbor client. However, these functions also provide a great opportunity to change the structure of the data sent to and from the Terraform schema. For example, the Harbor provider ignores the creation time field, as mentioned earlier. By ignoring this field in the provider code, the Harbor client can accurately mimic the API it interacts with without exposing unnecessary information in the Terraform resources.
Publishing and Using the Provider
With the release of Terraform 0.13.0, Hashicorp created a standard method for deploying provider plugins to the public Terraform Registry, as well as a corresponding method to allow users to import providers from the registry and use them in their own Terraform.
Once the Harbor provider was working, we published it to the registry. We restructured our existing documentation to match Hashicorp’s format to make the provider as easy to use as possible. We also published a GitHub release, including GPG-signed checksums for each released binary. We used GoReleaser to automate these steps, which greatly simplified the release process. The Terraform Registry validates the checksums and signatures and rehosts the GitHub releases.
Once the Harbor provider was published to the registry, Hashicorp included a basic example of how to import and use the provider. By adding just a few lines of code to your Terraform code, you can start using our Harbor provider yourself.
Extending Terraform to Manage Anything
A new provider can simplify the management of any obscure tooling that your organization uses, and you can even create a provider for internal tools. Extending Terraform with a provider plugin has helped enormously with Liatrio’s Harbor configuration. In the past, we managed Harbor with custom scripts, which couldn’t handle state or locking and that could create problems. By creating our own provider plugin, we can now handle state and locking from Terraform and define our configuration as code, even without an official provider.
Once you develop an understanding of providers, creating one is pretty easy. If you’re interested in creating your own provider, check out Hashicorp’s excellent documentation and tutorials on the subject or reach out!
