Managing Terraform State with Terragrunt

Terraform is still a popular tool for teams deploying infrastructure on the cloud. Its simplified templating language and cross-cloud support have made it a go-to for everyone from startups to enterprises.

But recently, a Terraform add-on has been gaining a lot of attention. In this article we'll dive into Terragrunt, the value it provides, and whether your team should use it for its Terraform workloads.

What is Terraform state?

Terraform by Hashicorp is a templating system for Infrastructure as Code (IaC) deployments in the cloud. Application teams can use its common syntax to script infrastructure deployments across multiple clouds. This includes not just the Big Three (AWS, Azure, Google Cloud) but other cloud providers such as Alibaba and Oracle.

In order to work its magic, Terraform needs to record certain information about your cloud account. Recording this information helps Terraform by:

Mapping IDs of resources in your cloud account to Terraform code
Storing metadata about your deployments
Optimizing performance of Terraform when used with large cloud accounts containing many resources

The sum total of this information is a given deployment's Terraform state.

How a Terraform state file works

Terraform stores state in a JSON file. This file can reside either on a local disk or in remote storage for team access.

To see the Terraform state file in action, let's look at the first tutorial in the Terraform docs. This uses a simple Terraform file to deploy a single Amazon EC2 instance onto Amazon Web Services (AWS). (Syntax highlighting provided by the Terraform extension for VS Code by Anton Kulikov.)

After initializing our connection to AWS as discussed in the tutorial, we can deploy this code using the terraform apply command.

After the command has run, you'll have a new EC2 instance in your AWS account. Additionally, you'll notice a new file, terraform.tfstate, in the directory where you saved your Terraform file.

The file contains a complete description of the EC2 instance you deployed. This includes the last state of all of its known configuration parameters, such as its network information, AMI configuration, and current running state.

Saving this state reduces the number of calls Terraform needs to make against your AWS account to obtain information. Excessive calls to the AWS API would slow down Terraform deployments. The slowdown would be even more noticeable in larger deployments, as Terraform would easily run up against AWS API call rate throttling limits.

Challenges for managing Terraform state

Sounds easy, right? Well...in theory. In practice, as usual, things are a little bit complicated.

The first challenge is finding the best way to manage remote state. As Gruntworks founder Yevgeniy Brikman notes, the Terraform state file contains sensitive information such as database passwords. Teams need to ensure they're storing it securely (i.e., privately stored and secured, encrypted at rest, etc.).

The other challenge is keeping your Infrastructure-as-Code templates DRY. As we discussed previously, "DRY" (Don't Repeat Yourself) is a design principle that encourages factoring out code into reusable modules and maximizing reuse. Unfortunately, as designed, Terraform's limited support for variables and the manner in which it handles state encourages code duplication.

Finally, another challenge (but by no means the last one) is managing multiple AWS accounts. Many application teams and vendors use multiple accounts to achieve isolation between environments and between different customers. You can juggle multiple accounts using native Terraform. But it's not straightforward and each solution involves nasty trade-offs.

How Terragrunt enables better state management

Enter Terragrunt.

Terragrunt is an open-source tool from Gruntworks that acts as a thin wrapper over Terraform. Its primary goal is enabling DRY architecture. In other words, it provides several features that aim to cut down on the amount of code duplication that using Terraform can create.

To understand the value proposition of Terragrunt, imagine that you have a full DevOps pipeline with multiple stages: Dev, Stage, QA, and Prod. Each deployment needs to use values that are unique to its environment. For example, you will likely use larger EC2 instance sizes in Prod than you do in Dev, Stage, and QA.

In a normal Terraform deployment, you'd need to copy your Terraform templates four times and make the required changes. This results in a lot of code duplication.

Terragrunt solves this by defining a new file format, an .hcl file, that you use in lieu of .tf files. .hcl files consist of the following parts:

A terraform block that defines how Terragrunt consumes Terraform. You can use this section to refer to Terraform files or modules that are stored in a remote location - e.g., in a team code repository.
A generate block that creates a configuration file. You can use this configuration file to define the cloud platform you're deploying into. You can also supply any necessary parameters such as target region or authentication variables.
An input block. This defines the parameters and values that are supplied to the Terraform code that you referenced earlier in the terraform block.

To use Terragrunt, you factor all of your Terraform code into reusable modules. For example, you can have separate modules for defining a VPC, Amazon EC2 instances, and your application deployment. You then check this code into a remote location, such as a Git repo.

With this infrastructure in place, you can create .hcl files for each of your infrastructure's components for each of your four stage deployments. The .hcl files refer to the common Terraform code in your repo. In other words, you use the same Terraform code for each stage deployment - the only thing that differs are the parameters you pass in via your stage-specific .hcl files.

Terragrunt use cases

As noted, the main Terragrunt use case is creating DRY Infrastructure as Code deployments. But Terraform provides a number of other critical use cases. Some of the most useful include:

Maintain DRY remote state configurations. Reduce duplication in your remote state storage by defining common configuration state values once at your project root and reusing them across environments.

Run Terraform across multiple files at once. terraform apply deploys one .tf file at a time. That's obviously not very efficient. If you've refactored your code into multiple Terraform modules, you'll want to execute them in series. Terragrunt includes a run-all command that enables you to run Terraform commands, such as validate and apply, across multiple Terraform modules.

Deploy infrastructure easily across multiple AWS accounts. Terragrunt can assume an IAM role on your behalf, drastically cutting down the amount of work required to manage multi-account deployments with Terraform.**.

Is Terragrunt right for your system?

So should you use Terragrunt?

On the one hand, Terragrunt has a lot of excellent features. It solves problems that many teams that use Terraform extensively will ultimately run up against. From that perspective, it's worth investigating how your team can benefit from it.

One the other hand, Terragrunt adds extra complexity to your development process. If your team is just starting with its use of Terraform, layering Terragrunt on top of it immediately might prove overwhelming.

This is a similar situation to considering whether to use AWS CloudFormation or the AWS Cloud Development Kit (CDK). The CDK produces CloudFormation code, which means it's best used by devs and teams that already have some base experience with CloudFormation.

In the same vein, I'd recommend looking at Terragrunt - but maybe after your team has its first few Terraform deployments under its belt.

TinyStacks and Terraform

Tools like CloudFormation and Terraform/Terragrunt make it easy to deploy infrastructure repeatedly and at scale. At TinyStacks, we take Infrastructure as Code a step further.

Our tools enable our customer's development teams to quickly and easily deploy application stacks to any DevOps stage with the click of a few buttons. Using TinyStacks, your teams can assembly deployments from modular IaC templates and deploy to the cloud in minutes. Monitor all deployments company-wide from a single pane of glass.