DRY Infrastructure as Code: How Not to Repeat Yourself

DRY Infrastructure as Code: How Not to Repeat Yourself

What is DRY? And how can you apply it to your cloud infrastructure?

Jay Allen's photo
Jay Allen
·

8 min read

The acronym DRY - Don't Repeat Yourself - has been a go-to principle in software for years. It's also important, not just in application code, but in your infrastructure templates as well. In this article, I lay out what DRY infrastructure as code is, and discuss concrete tactics for achieving infrastructure modularity and reuse across your projects.

Infrastructure as code: the promise - and challenges

The cloud has brought a lot of changes to the software industry. Infrastructure as code (IaC) is one of the most revolutionary. Instead of standing up servers, storage, and networks manually, dev teams can specify their entire application as IaC. All of the major cloud providers even support their own declarative syntax for defining IaC, such as AWS CloudFormation and Microsoft Azure Resource Manager Templates.

The benefits of IaC are numerous:

  • Deploy infrastructure repeatedly - eliminate the guesswork from deployment
  • Maintain infrastructure and configuration in source code - ensure all changes are code reviewed and tracked
  • Stand up multiple environments - easily support a staged deployment pipeline for new application changes

However, achieving these benefits requires legwork. And it's easy to make short-sighted decisions that make maintaining your infrastructure as code projects harder.

At the top of that list is infrastructure code duplication. Once you have a working dev deployment, for example, it can be tempting to just copy and tweak it until it also works in production. Other teams at your company may also copy code out of your source repository and mold it to suit their own purposes.

However, such duplication quickly becomes costly.

Leandro Marty at MixMax documents how his own team had separate staging and deployment code bases. At first, it worked well. Over time, the process started breaking down. The team found itself forced to make changes to infrastructure in multiple locations. Staging and production started to drift apart.

In other words, it was a real mess.

Why infrastructure as code should be DRY

Creating a solid Infrastructure as Code foundation is the backbone of creating a solid DevOps pipeline.

With good infrastructure as code templates, you can create a full deployment pipeline that moves your application through multiple stages - e.g., dev, test, staging, production. You can vet your application at each stage, ensuring the changes you're deploying will work in production.

However, if you're not maintaining DRY infrastructure as code, then your DevOps effort may crash shortly after takeoff.

Consider MixMax's experience above. They only had two environments - staging and production. Now imagine you're maintaining separate infrastructure code for three or more environments. Every time you need to make a change to your infrastructure, you increase the risk of drift - i.e., the risk that the environments will get out of sync with one another.

If your environments aren't near copies of one another, you can't be ensured that you're testing the same thing in test that you'll run in production. This sabotages one of the main benefits of a DevOps pipeline: vetting your changes prior to release. Suddenly, you're back right in the Bad Old Days of application development where each of your environments were distinct physical deployments with little coordination between them.

The security risks of non-DRY code

Duplicated infrastructure as code also risks opening you up to security holes. Suppose, for example, you discover that your production application is leaving open SSH or RDP ports on certain virtual machines.

So you close it up...in production. But if you neglect to make the same change to all your environments, you risk an intruder gaining access to your private networks. Depending on the rights assigned to the machine, you could also expose access to secrets or even enable an intruder to spin up cloud services in your account.

Compounding the problem

Finally, suppose you work at a company with multiple independent application development teams. Ideally, you want each team using the same standards to deploy cloud infrastructure. This means creating a set of reusable, customizable templates that set standards for basic infrastructure components such as networking, storage, and compute.

Unfortunately, in today's fast-paced market, such work often falls by the wayside. Different teams often design and deploy their own infrastructure.

While there are some agility benefits to this, it leads to a lack of consistency between projects. It also makes it more difficult to stand up new functionality. In order for a new team to get started, they have to reinvent the wheel and build their own infrastructure from the ground up.

How TinyStacks is enabling DRY solutions

At TinyStacks, we've been working to enable DRY Infrastructure as Code deployments for years. TinyStacks is an Infrastructure as Code delivery platform that enables teams to launch their applications in hours, not months. Define your infrastructure once and let application teams deploy their apps safely and predictably via a self-service model.

You can check it out today for yourself and see how easy it is to deploy your application on TinyStacks.

How to achieve DRY infrastructure as code

So how do you get DRY (besides using TinyStacks, of course)?

Sadly, it's easier said that done. It takes time, testing, and coordination across multiple teams. But the effort pays dividends in terms of reduced time to market, higher quality deployments, and more consistent and secure infrastructure.

Below are some of the foundational practices for developing DRY infrastructure as code templates.

Parameterize your deployments

The first step to a DRY infrastructure as code deployment is removing any hard-coded parameters that tie your application to a specific deployment stage. In other words, you want to parameterize anything specific to a given deployment.

Declarative IaC language like AWS CloudFormation support parameters - and you should use them liberally! For example, you can use parameters to include the name of the current deployment environment in your deployment:

Parameters:
  Environment:
    Type: String
    Default: dev

You can then use the parameter to give your resources a unique name depending on their stage. This is useful if you're deploying multiple versions of your stack into the same cloud account or subscription: 

Ec2Instance: 
  Type: AWS::EC2::Instance
  Properties: 
     ...
              Tags:
        -
          Key: Name
          Value: !Join [ '-', ['EC2Instance', !Ref Environment] ]

Another use for parameters is to parameterize references to dependencies that will differ between environments. The best and most secure way to do this is using a secrets store such as AWS Secrets Manager. You can pass the resource name of a secret into a template as a parameter and then resolve it dynamically inside of your template, as in this example using AWS RDS:

  MyDBInstance:
    Type: 'AWS::RDS::DBInstance'
    Properties:
      ...
      MasterUsername: !Sub '{{resolve:secretsmanager:${SecretsManagerTutorialAdmin}::username}}'

Microsoft Azure's Bicep, a new language for generating Azure Resource Manager templates, makes such parameterization easy using the existing keyword, which you can use to refer to an existing keyword and reference its properties:

param sqlServerName string
param adminLogin string

param subscriptionId string
param kvResourceGroup string
param kvName string

resource kv 'Microsoft.KeyVault/vaults@2019-09-01' existing = {
  name: kvName
  scope: resourceGroup(subscriptionId, kvResourceGroup )
}

module sql './sql.bicep' = {
  name: 'deploySQL'
  params: {
    sqlServerName: sqlServerName
    adminLogin: adminLogin
    adminPassword: kv.getSecret('vmAdminPassword')
  }
}

Modularize the components of your infrastructure

Parameterization is a great first step. Using thoughtful parameterization and a secrets store, you can create a single set of IaC deployment templates that can deploy any environment for your application.

The next step is to design components for reuse across multiple projects through modularization. Using modularization, you can create basic infrastructure components - such as virtual networks, storage locations, Docker container registries - that can be leveraged across multiple projects or even totally separate teams. Such modularization has multiple benefits:

  • You can start new projects more quickly - no more reinventing the wheel.
  • You reduce the learning curve. Application engineers with little IaC experience can get started deploying apps more quickly.
  • You create standards for security, performance, and other best practices that other teams can adopt with little overhead.

There are a couple of ways you can modularize your deployments. One is to create chained stacks of IaC templates, using parameters to tie them together. For example, you could define an AWS Virtual Private Cloud (VPC) - complete with public and private subnets and NAT Gateways - in its own standalone template. Then you could have a second template for the infrastructure that's specific to a given application.

The other is to use the module functionality built into your cloud provider's IaC system. Both AWS and Azure Resource Manager (both JSON and Bicep templates) support defining reusable components as modules that can be used across numerous templates.

Make reusable infrastructure components discoverable

Reusable components aren't useful unless developers in your company can find them.

The best place to place reusable modules is the same place you place any source code: under a source control system, such as Git. By placing all reusable modules in a central Git repository, you can track changes as well as foster cross-team collaboration via pull requests.

You can also use a module registry as a repository for available modules within your organization. AWS's CloudFormation modules have a built-in method for discovery, as all modules used in a template have to be registered in the CloudFormation registry in the region in which they'll be used. Azure also supports the concept of a private registry for storing and discovering modules. Both registries support versioning and tagging so that consuming teams can control which versions they pick up for their projects.

Conclusion

Moving your teams to a DRY Infrastructure as Code model isn't easy. But the work involved pays off in the long run in increased quality, enhanced cross-team security, and faster time to market.

Next steps

To see how TinyStacks is redefining DevOps and IaC, sign up today and give it a test-drive!

DevopsAWSCloudCloud Computing