Developers are constantly seeking ways to enhance their workflows and streamline the development process. One such method has gained popularity in the Go programming language: the tools.go pattern. It can be found in some of the most popular open source projects on Github like Kubernetes, Open Telemetry, and thousands more. Clearly, the tools.go pattern is well regarded and utilized but still has some challenges to overcome.
The tools.go Pattern and its Merits
The pattern leverages the tools.go and go.mod files to manage various tools required for the project. This provides a number of advantages like defining the toolchain as code, building the tools from source with simple commands, using Dependabot or Renovate to automatically manage toolchain versions, etc. The beauty of this pattern lies in its simplicity and effectiveness in keeping development dependencies separate from production code.
Parsing tools.go... With Bash?
Let's look at Open Telemetry implements the tools.go pattern to manage tools in the opentelemetry-collector-contrib repo:
Don't spend too much time trying to decipher this Makefile wizardry; We skipped a lot of lines. There are two important details to note. First, the tool names are extracted from tools.go by running grep and tr. Second, these tool names are just passed to the tools.go command. Wait, really? Isn’t that a hack use of mod.go?
The pattern suffers from a discoverability problem as well. It’s generally not well documented outside of specific implementations. You can find several blogs describing the pattern but not many and there’s no standard implementation. Currently, the best source of information is to study and copy an existing implementation.
At this point, you may be asking why these important and professional projects are using a method like this. Don't we have a better way? The reason is simple: It works and it works well. Just don't look at it or make changes!
And no, we don't have a better way. At least, not yet.
Enter the Go Proposal: A Solution in the Making
Thankfully, the Go community is addressing this problem. Two years ago, Conrad Irwin introduced a proposal that aims to address the challenges associated with with the tools.go pattern. This proposal, titled Adding tool dependencies to go.mod, outlines a comprehensive plan to enhance the tooling ecosystem in Go by embracing the tools.go pattern directly in the go command.
The proposal describes two significant changes. First, the go.mod file will allow tool directives to define tool dependencies. This basically moves the contents of tools.go into go.mod. A welcome change considering that currently, we never actually compile tools.go. Second, the go command will be extended to support a new tools metapackage which will supply the following new go subcommands:
Forget the Makefile implementation. These proposed commands are dramatically simpler and offer all the same benefits.
This also addresses an ugly bootstrapping issue. Currently, most projects utilizing the tools.go pattern rely on a Makefile to define install commands. But what if make isn't available? By moving this functionality directly into the go binary, the toolchain can be installed without needing a Makefile to record complex install commands. Perhaps we can soon enjoy some Makefile alternatives!
Progress and Development
Excitingly, the Go community has accepted the proposal and the features are being developed in Go 1.22! Considering that Go 1.22 is very near its release, we don’t expect to see Tool Dependencies released until Go 1.23 or later. Whenever it comes, this new feature mark a significant step forward in providing a seamless and standardized experience for managing tools and dependencies in Go projects.
As we eagerly await the integration of the Go Tool Dependencies proposal, the tools.go pattern is set to evolve from a clever hack to a fully-supported and standardized feature in the Go programming language. This enhancement promises to make the lives of developers easier, fostering a more efficient and reliable development process. By anticipating these upcoming changes, developers can look forward to a future where managing tools and dependencies in Go projects is smoother, more consistent, and tailored to the needs of modern software development.
