Why not use Terraform to manage your Terraform?

TLDR

When working with a production Terraform configuration, there will inevitably be a time when you outgrow the single workspace pattern. Whether you split workspaces by account, region, service, or something else, soon you’ll find that you need to break your Terraform configuration apart.

An aside on the term workspace:

The word workspace in this post follows the Terraform Cloud and Scalr definition instead of the Terraform CLI definition. Hashicorp has a blurb here: Terraform Cloud vs. Terraform CLI Workspaces, but it boils down to this:

Terraform CLI workspaces are infrastructure as code split into different directories and have different state files. Terraform Cloud and Scalr workspaces are similar in that they have individual state files; however, these tools also provide more features like specific role-based access control and variable configurations.

Also worth noting is that in Spacelift, the equivalent of a workspace is a stack.

The Terraform configuration I work with daily consists of 31 active workspaces and 36,747 resources as of writing this. Two of those workspaces have over 9,000 resources a piece, and as you can imagine they take quite a bit of time to run. Logically breaking these workspaces down into smaller chunks will help speed those Terraform plans and applies up. It also helps unlock safe developer inclusion since I can allow an application team access to a workspace that contains only their infrastructure. All of a sudden, one of my 9,000 resource workspace turns into 12 workspaces that have 750 resources each with the added benefit of my engineering teams being able to iterate faster and independently of each other.

Of course, if I’m managing fifty or even thirty workspaces, I probably have repeated variables across them. The seed workspace lets me manage all of these variables (including sensitive ones) via code. I also can manage teams, tags, VCS configuration, and more.

The pattern lets me leverage Terraform patterns to maintain consistency and state on a critical layer of my tooling. The three demo repos I created are all very straightforward and should be easy to comprehend. The primary difficulty ends up being the use of the providers isn’t as thoroughly tested as some of the primary Terraform providers like AWS or AzureRM so the documentation ends up having quirks. For example, in the Terraform Cloud demo I originally created teams for a more fleshed out sample, but the provider documentation for the tfe_teamresource doesn’t mention that the capability isn’t available in the tier I was using.

In my Terraform Cloud Workspace demo, you can see I define four key values. The values for the keys ultimately live in Terraform Cloud’s variable section which I can lock down using the built-in access controls. But since those values are a part of my seed workspace, I can assign them to the workspaces I create programmatically which makes rotation a breeze as well.

The fifth variable I define is actually a global Terraform version that I apply to each of my workspaces. This is just an example. I could do the same with branch name, SSH key, etc. This pattern is all about keeping consistency in a repeatable and secure method.

variable "terraform-version" {
  type        = string
  description = "Global Terraform version to use"
  default     = "1.1.4"
}

variable "azure-dev-key" {
  type        = string
  description = "Development key for Azure"
  default     = ""
}

variable "azure-prod-key" {
  type        = string
  description = "Production key for Azure"
  default     = ""
}

variable "aws-dev-key" {
  type        = string
  description = "Development key for AWS"
  default     = ""
}

variable "aws-prod-key" {
  type        = string
  description = "Production key for AWS"
  default     = ""
}

Managing variables for your Terraform tooling with Terraform ends up carrying a lot of overhead. Hashicorp has introduced something named variable sets to Terraform Cloud. In Scalr, you can do something similar with shell variables in your account dashboard or in your environment configuration. It’s a different way of doing the same thing and works just as well as what I did above.

The Cost

There is a downside to this pattern. If you are just getting started or only need a fast prototype or change, why would you want to go through opening up a PR against your seed workspace and running it through Terraform? It’s certainly much faster to create a new workspace and wire it up with the existing VCS connection. The only answer I can provide is consistency. If you are a team of yourself or just two or three, this pattern makes less sense. But once the use of Terraform is growing and you begin to have tiers of access and experience levels with your Terraform usage, perhaps this pattern makes sense.

References