Compiling a custom Terraform provider

Several times now, I’ve been presented with provider bugs in Terraform - specifically the AWS provider - that are fixed in PRs that have sat unmerged for months, if not years. While Hashicorp has now codified that community pull requests aren’t being reviewed, I’ve had a dirty trick up my sleeve for about two years now: in some projects, I don’t use the upstream provider anymore. Instead, I run a fork, where I’ve cherry-picked in any fixes I’ve needed. And after having admitted to doing that publicly, I got an e-mail just yesterday, asking how I did it.

Truth be told, I’ve never wanted to blog about my solution, because it feels dirty. At best, it’s a workaround for what should be a more seamless community process, but I’ll get off my soapbox and just give the details of what I’m doing. It’s actually pretty simple:

1. I fork the provider code, and use git cherry-pick to cherry-pick in the commits I want from various other PRs. (I prefer doing it this way, capturing the original commits, so I can keep track of what’s on my fork and where it came from.)
2. I hijack the version string 0.0.1 - which, luckily, has never had a real release. It doesn’t exist upstream, which is a happy accident.
3. When I run terraform init, I use Docker to compile the provider - optionally cross-compiling if I’m running Terraform locally on a Mac, rather than in CI on Linux.
4. I put the compiled provider in the plugin cache directory where terraform init would normally put it.

That’s it. Here’s a very basic script that can run inline with CI - or, alternatively, be run in a wrapper script that also calls terraform init:

if [ "$(uname)" == "Darwin" ]; then
  export GOOS=darwin
else
  export GOOS=linux
fi

PROVIDER_DIR="$(pwd)/.terraform/plugins/$GOOS_amd64"
PROVIDER_FILE=terraform-provider-aws_v0.0.1_x5
echo "Building custom AWS provider..."

if [ ! -d $PROVIDER_DIR ]; then
  mkdir -p $PROVIDER_DIR
fi
if [ ! -f $PROVIDER_DIR/$PROVIDER_FILE ]; then # custom provider has no version string
  buildroot=$(mktemp -d)
  git clone --depth=1 git@github.com:my-copy-of/terraform-provider-aws.git $buildroot
  docker run -it --rm -e GOOS=$GOOS -v $buildroot:/buildroot golang:1.14 bash -c "cd /buildroot && make tools && make build && mv /go/bin/$GOOS_amd64/terraform-provider-aws /buildroot/$PROVIDER_FILE"
  mv $buildroot/$PROVIDER_FILE $PROVIDER_DIR/
  rm -rf $buildroot
fi
if [ ! -f $PROVIDER_DIR/$PROVIDER_FILE ]; then # if it still doesn't exist, it failed to build and we should exit
  echo -e "Custom provider failed to build; refusing to continue."
  exit 1
fi

Note that you’ll also need to set GOARCH if you’re working across architectures (for instance, if you have ARM anywhere in your ecosystem, like on an M1 MacBook). I don’t have this problem (yet, but I’m always hopeful that ARM will only become more ubiquitous), so I didn’t solve for it here.