Guide to AWS DevOps Automation

I've talked a lot about the importance of DevOps automation. And I've even covered many of the tools that are generally available to assist you on your journey.

In this article, I want to focus on companies and teams that are specifically starting their DevOps automation journey on AWS. Specifically, I'll talk about how you can leverage the tools and technologies available on the AWS platform to create a fully automated application deployment pipeline.

What is DevOps automation?

I've gone into detail about DevOps automation in other articles. So I won't belabor the point. Here's a quick summary for those who may have missed it.

The goal of DevOps is to ship new versions of software applications with high reliability. This means collapsing the divide between our applications and the environments in which they run. In DevOps, our applications and our environment are bundled together into the final deliverable.

Using DevOps automation, we can create a new, production-ready version of our application every time we make a significant change - whether major or minor - to our app and its environment. We can use DevOps automation at every step of the application deployment process:

• Using Continuous Integration (CI), we can commit, run tests, and perform security checks against our source code every time we check in a change to source control.
• Using Continuous Delivery (CD), we can create a new, deployable package - including the application and its runtime environment - from every check-in. You can then use Continuous Deployment (a.k.a. "the other CD") to ship your code to your dev, test, and production environments.
• Using Infrastructure as Code (IaC), we can stand up all of the components - networks, servers, storage, APIs, data processing - required to support our application in production.
• Using monitoring, we can gauge the quality of our changes at runtime - and raise timely alerts should anything go wrong.

Why automate DevOps?

The key question isn't "why automate DevOps". It's "why wouldn't you automate DevOps?!" Deploying even a simple application means standing up both your application and all its accompanying infrastructure. If you're following a DevOps model, you're standing up multiple environments across several deployment stages.

Can you do all of this manually? Maybe. But you greatly increase the chance for human error. That means botched deployments and system downtime.

In other words, "DevOps" and "automation" practically go hand in hand. Using automation, you can ensure your DevOps-driven deployments are rapid, repeatable, and of high quality.

The AWS DevOps automation story

So why use DevOps on AWS?

From the beginning, AWS has built automation into its platform. Yes, you can create your application entirely from the AWS Management Console. However, AWS has always strongly recommended that customers take an "automation-first" approach when deploying applications on the service.

To make this easier, AWS has invested heavily in tools that enable Infrastructure as Code (IaC). With IaC, development teams can define all of the infrastructure their application needs to run in source code. This enables teams to create, destroy, and re-create their entire application environment repeatedly, reliably, and at will.

The most famous example of AWS's commitment to IaC is AWS CloudFormation. CloudFormation is a declarative language with which you can create an entire architecture for your application. Nearly every AWS feature - from virtual networks and servers on up - can be instantiated via CloudFormation.

This dedication to automation continues with the AWS DevOps story. Using AWS DevOps tooling in combination with IaC technologies such as CloudFormation, you can drive a new version of your application into production with every source code check-in.

Use cases for AWS DevOps automation

In my previous article on DevOps automation tooling, I talked about different toolsets available from multiple different companies. Stitching all of these together into a single DevOps deployment pipeline requires time and testing.

By contrast, AWS has designed its DevOps tools to integrate with one another. This means you can spend less time connecting DevOps solutions from multiple vendors and more time focusing on your application.

In recent years, AWS has made their DevOps story more transparent by unifying many of these tools under a single interface. Unfortunately, it's a bit hidden. You won't find a "DevOps" menu item anywhere in the AWS Management Console. You can navigate to it by searching for a feature that's part of the AWS DevOps stack. In this case, I searched for CodeCommit.

Using this visual roadmap, let's walk through each of these features of the AWS DevOps automation story one by one and see how they build upon one another.

AWS DevOps automation for building apps

The first step in your DevOps journey is building your app. In a DevOps model, your team will store most project files - including application and infrastructure code - in a source code repository.

AWS CodeCommit: Storing source code

Git is pretty much the de facto standard in source code control. Many projects use a Software as a Service-style hosting service such as Github or Gitlab.

If you're all-in on AWS DevOps, however, the company has a solution for you. AWS CodeCommit is Amazon's own hosted Git service. Using the Management Console, you can easily create a new repository for your project's code. You can then use AWS Identity and Access Management (IAM) to give your development team permissions to clone the repo and check in changes.

As with services like GitHub, you can use CodeCommit to manage developer check-ins and mandate best practices such as code reviews. You can combine CodeCommit's Pull Requests (PRs) feature with approval rule templates, for example, to require a minimum number of code reviewers before committing a check-in to a branch.

AWS CodeArtifact: Storing Outputs and Dependencies

Code isn't the only thing you need to manage, of course. Your projects will likely need to pull in dependencies from public repositories, using tools such as NPM and NuGet. You'll also want somewhere to store the outputs of your own builds for the next stage of your deployment process.

Enter AWS CodeArtifact. Using CodeArtifact, you can define a location to store your project builds. You can also create an artifact stream for your project's dependencies. This ensures that your team is always pulling its dependencies from known, reliable sources.

CodeArtifact also provides a convenient place to share the outputs of your project with other teams. For example, you could grant other teams in your company access to your libraries so that they can incorporate them into their own projects. In other words, your own project can serve as an upstream artifact repository for another team.

AWS CodeBuild: Building and testing your apps

Whenever you check in source code to CodeCommit, you can set up a Continuous Integration process that performs any or all of the following tasks:

• Lints and builds your application source code to ensure it can compile (for languages that require compilation like Java, C#, or Rust) and is free of basic errors
• Runs unit tests that check that your source code works and is functionally sound
• Runs security scanning tools that perform sanity checks, such as ensuring you aren't checking in credentials directly to source code

With AWS CodeBuild, you can build the code in your CodeCommit repo and save the output in an Amazon S3 bucket. You can set up different builds for different Git branches or customize a single build to build outputs from multiple branches.

Both CodeCommit and CodeArtifact are easy to set up using the AWS Management Console. To use CodeBuild, you'll need to author a BuildSpec file. The BuildSpec is a YAML file that specifies which commands to run as part of your build. AWS then provisions and runs your build on one of its pre-built Docker containers. (You can also supply your own Docker container if you need a highly customized build environment.)

Getting the BuildSpec YAML correct can require some trial and error. Fortunately, AWS maintains a set of example BuildSpecs upon which you can build.

At its most basic, you can use your buildspec.yml to compile and create a deployable artifact for your application. You can also use the buildspec.yml to deploy your application directly from the build to a number of AWS services, including Elastic Container Service (ECS).

AWS DevOps automation for deploying apps

As noted above, you can use CodeBuild, not just to build your application, but to deploy it. However, this will probably only work for a few teams with the simplest of scenarios.

The next two technologies in the AWS DevOps stack provide a more comprehensive approach to deploying applications. And one of them will enable us to bring all of these separate solutions together into a single, comprehensive deployment pipeline.

AWS CodeDeploy: Orchestrate a progressive rollout

First up we have AWS CodeDeploy. With CodeDeploy, you can deploy your application using a number of rollout strategies. CodeDeploy supports multiple methods of deployment:

• Application packages hosted directly on Amazon EC2 virtual machines
• AWS Lambda functions
• Docker containers hosted on an Amazon ECS cluster

To use CodeDeploy, you need to define two entities: an application and a deployment group. The application defines your preferred compute model: EC2, Lambda, or ECS. The deployment group defines where to find your application artifacts and the method you want to use to roll them out.

The rollout methods available via CodeDeploy depend on the compute model used. For example, if your application is an AWS Lambda package, you can replace all running Lambda instances immediately with your new code. Alternatively, you can specify a rolling deployment.

CodeDeploy Lambda supports two models: canary, which swaps out a percentage of your fleet with your new code and then replaces the rest a set number of minutes later; or linear, which swaps out a percentage of your fleet with the new code every number of minutes.

CodeDeploy's greatest strength is that it makes implementing otherwise complex deployment strategies simple.

AWS CodePipeline: Orchestrating a full, multi-staged deployment

So far, each of the technologies we've looked at does a single specific job in the DevOps process. What we need is something that can tie all of these elements together. We want check-ins to our source code repository to create our entire application stack. In other words, we want an application deployment pipeline.

Enter AWS CodePipeline. You can configure a pipeline in CodePipeline to connect a branch in your CodeCommit repo and build it with a CodeBuild build configuration.

Then, you can configure how you want to deploy your application. CodePipeline supports over 10 different methods of deployment. The most powerful of these is our old friend CloudFormation, which you can use to define the infrastructure and runtime environment for your application prior to deployment. You can select a CloudFormation file that you've checked in to your code repository and supply it with parameters that point to your build artifacts.

You can customize your pipeline at each step of the process. This enables you to create a pipeline for each stage of your deployment process. For example, let's say you want to have separate environments for dev, test, and production. You could accomplish this by:

• Building each pipeline off of a separate branch in CodeCommit
• Supplying environment variables to your CodeBuild project to label and tag build artifacts according to their stage
• Creating different configuration files to your CloudFormation templates to create separate infrastructure components for dev, test, and production

AWS DevOps automation for monitoring apps

As I discussed previously, monitoring is essential to DevOps automation. Detailed and extensive monitoring enables you to see in an instant whether a given deployment is healthy or unhealthy.

Amazon CloudWatch

The heart of AWS's monitoring story is Amazon CloudWatch. CloudWatch provides two features that are indispensable to any application development team: metrics and logging.

Using CloudWatch Metrics, application developers can track the performance of their systems in real time. AWS supports emitting a core set of metrics automatically from each service - e.g., CPU and memory utilization from Amazon EC2 instances, or request timeouts from AWS Lambda functions. Developers can also implement custom metrics and emit them directly from their applications.

With CloudWatch Logs, software teams can capture detailed logging output from their AWS services and their custom application code. Team engineers can use the output from logs to debug issues in their running environments.

Benefits of AWS Automation

There are numerous benefits to implementing DevOps automation on AWS:

Speed of deployment. It's possible to create and manage your apps solely with the AWS Management Console. (When your project is small- to medium-sized, at least.) But it's not particularly efficient. And manual maintenance leaves plenty of room for human error. Automating your AWS deployments ensures consistent, reliable, and rapid deployments.

Secure management. If you already host your applications on AWS, it makes sense to keep your DevOps story there, too. You can secure access to your DevOps tools using the same identities in AWS Identity and Access Management (IAM) that you use for securing access to your other AWS resources. Storing everything in AWS also means you have visibility into both your DevOps pipeline and your infrastructure in a single location.

Cost. Pricing is always a concern when hosting anything on AWS. However, the pricing for its DevOps services is comparatively cheap. You'll only pay $1/month per pipeline you create in CodePipeline, for example. CodeCommit supports five users for free; additional users are only $1/month.

Challenges of AWS Automation

However, while implementing DevOps on AWS has its advantages, there are also several downsides to consider before embarking down this path.

Learning curve. If you haven't done DevOps before, you'll have a lot to learn. AWS attempts to simplify certain concepts and get you up and running quickly. However, it's not always evident how to utilize the tools to meet your specific scenario.

Time investment. An automated DevOps pipeline is a thing of beauty. But, even on AWS, it takes a lot of trial and error to create and debug one to the point that it works reliably. If you haven't done DevOps before, the investment required will seem steep.

Hard to manage more complex deployments. The AWS DevOps tools make it easy to create deployments of simple to medium complexity. However, as the complexity of your application grows, you may find that things get harder to manage. This is especially true if you're maintaining multiple applications or services. Tools such as CodePipeline don't make it easy to assess the state of a complex application or application ecosystem at a glance.

Build faster without abstractions or undifferentiated DevOps tasks

We enable our customers' development teams to create a full DevOps pipeline even if they know little about AWS and DevOps. Just tell us where your code lives on GitHub or GitLab and we'll do the rest.