Best Practices for DevOps Automation

If it's not automated, are you really doing DevOps? The speed of business requires shipping software faster than ever. That means removing time-consuming and error-prone manual steps from our build and releases processes. In this article, I'll discuss what DevOps automation is, walk through some examples, and discuss the key best practices involved in moving from a manual to an automated release process.

What is DevOps automation?

As I've discussed in previous articles, DevOps is a set of best practices and tools for increasing the speed at which organizations deliver high-quality software.

The term "DevOps" originally developed from the systemic disconnect that traditionally existed between the people who built software - the developers - and the team that maintained the infrastructure on which it ran - the system administrators, or operations team. Developers would develop code that "worked in dev", only to encounter numerous errors - missing libraries, mismatched configurations - when it shipped to production.

DevOps aims to eliminate this mismatch by treating an application and the environment in which it runs as a single deliverable. DevOps automation enables this goal by reducing the release process to a set of Git check-ins or mouse clicks that activate a set of build and release scripts. This process is typically divided into two parts:

Continuous Integration (CI), which checks the integrity and functionality of new code checked in to a source code repository and verifies that a new release can be built.
Continuous Deployment (CD), which ships the built software onto new or existing infrastructure.

The goal of DevOps automation is to make software releases easy and error-free. The more automated your release process is, the less likely your releases will fail because someone mistyped a configuration variable, or forgot to migrate an encrypted secret into production. DevOps automation aims to take the guesswork out of releases and make them reliable, repeatable processes.

DevOps automation examples

Examples of DevOps automation abound - particularly with the advent of the cloud. Cloud services like Amazon Web Services (AWS) pioneered the concept of "Infrastructure as Code". Developers can use domain-specific languages like AWS CloudFormation or a major programming language to create the virtual machines, networks, databases, file stores, and other computing resources they need to run their applications automatically and on demand.

Infrastructure is just one of the many things that can be automated in DevOps automation. Others include:

Running test suites
Running linting tools to detect coding errors or enforce coding best practices
Compiling binaries
Ensuring security through security checks (e.g., checking binaries for known exploits, detecting secrets checked into source)
Signing binaries and scripts
Packaging and archiving of build artifacts
Deploying code onto virtual machines or Docker container clusters
Gradual shifting of traffic from the previous version of an app to a new version ("blue-green" deployments)
Running runtime tests, such as load/scale testing or API vulnerability checks
Generation of documentation (e.g., OpenAPI specifications)
Sending release emails

In other words, all significant parts of a software release process can be automated via DevOps automation.

DevOps automation best practices

How do you get started on the road to DevOps automation? Every company and every team will approach their journey differently. Here are a few guidelines on how to get started.

Develop a DevOps automation mindset

I've worked in numerous organizations as both an engineer and an engineering manager leading DevOps initiatives. And the one recurring phenomenon I see again and again is a cultural resistance to DevOps automation.

Some people are concerned that automating critical tasks means they'll be out of a job. Others express concerns around the time and effort it takes to automate tasks, especially when they feel their plates are already full.

A DevOps automation mindset requires support from managers and leadership. Leaders need to make it clear that the goal isn't to automate anyone out of a job. Rather, the mission is to increase the quality and velocity of releases - which will free up engineers to focus on writing application code.

Leaders also need to commit to a DevOps automation mindset by backing them with resources. This can mean freeing up existing personnel to focus on DevOps automation tasks or even hiring full-time DevOps engineers.

There's a common saying in the DevOps space: People over Processes over Tools. Instilling that mindset in your organization is critical to a successful DevOps automation effort.

Establish your CI/CD pipeline

A solid set of CI/CD processes, or a CI/CD pipeline, is at the heart of a DevOps automation strategy. A successful CI/CD pipeline will enable your team to push a new set of functionality to production simply by checking in source code to Git.

A well-established CI/CD process will:

Accelerate time to market for new features and critical bug fixes
Reduce manual errors by automating crucial steps in the deployment process, such as infrastructure and environment configuration
Reduce time spent on creating new deployments and debugging the resulting issues
Ensure a smooth transition for users from one version of the application to the next

To get started, teams should start by discussing how they want their CI/CD process to work. Some critical questions include:

How will we manage branches in our source code repository? I.e., what actions will trigger a deployment to production?
How many stages will our DevOps automation pipeline have?
What quality gates will we define? E.g., will we push changes to production automatically? Or will we still have manual approvals for production deployments?
What automated checks do we need to run to ensure quality and security?
How do we verify the quality of a release in production before rolling it out to users? Do we use a canary deployment, some form of blue/green deployment, or both?
How many regions do we roll our code out to, and in what order?

These are all large questions. Odds are your team won't be able to answer them all at once. It's a good idea, therefore, to tackle them in an iterative and incremental manner. For example, you can work on defining a branching strategy and building out the tools and processes you need to automate builds first. From there, you can work on deploying your code automatically to a developer environment. You can then add more stages gradually to your pipeline until you have a full, working production pipeline deployment.

Use flexible toolsets

You'll likely develop your overall DevOps automation strategy while also experimenting with the various tools and platforms that will transform it into reality.

While selecting your tools, it's important to choose ones that are flexible, as opposed to tools that lock you into a specific methodology. In particular, you should ensure that your tools can grow and change as your business grows.

A lot of development teams get hit by this when they adopt Platform as a Service (PaaS) solutions like Heroku. Initially, when their application is small, the PaaS works very well for their needs. However, as the complexity of their deployment grows, the limitations of a PaaS - and the cost - quickly catch up with them.

Good DevOps automation tools - like Jenkins - provide an open-ended framework that enable DevOps teams to start out small but eventually scale out to large, complex multi-region cloud deployments.

Create a rich monitoring solution

So your code is in production. How do you verify that it's working as expected? Manual testing is, and will likely always remain, part of the verification process.

But manual testing has its limits. Manual tests won't alert you to problems that don't manifest until hours or even weeks after you ship a change. And a manual strategy quickly breaks down as your application suite grows.

For these reasons, a robust monitoring solution is essential to a DevOps automation strategy. Your team should define early on what it's going to monitor, how they'll transmit that data, and how they'll track and visualize it.

What and how you monitor is up to you and your team. However, many DevOps practitioners recommend a few best practices:

Develop a "monitor everything" mindset in your team

Application errors, latency, uptime, virtual machine CPU and memory consumption, unauthorized access attempts - if it can be detected and monitored, it should. The more signals your team has, the easier it will be to ensure ongoing application health.

Monitor early

Adopt a "shift-left" testing approach, in which monitoring and testing are built into your application development process from the outset. This will give your team the confidence that it's hitting the quality goals required to ship code to production.

Integrate with internal issue management systems. Critical errors - such as an unexpected number of 500 server errors or repeated unauthorized access attempts - should trigger Sev 2 or Sev 1 issues automatically in your organization's issue management system. By raising critical issues automatically, your team can catch small problems before they become major incidents.

Define a change management strategy

Whether it's a new feature, an important customer request, or an urgent security fix, your team needs a defined process for rolling out application changes. Change management dictates the procedure for scheduling and deploying a change with minimal impact to existing customers.

One of the most important parts of change management is defining how a change will be assessed, accepted, and tracked. New requests need to be weighed and balanced against the existing backlog of scheduled work and planned accordingly. Many DevOps-driven organizations use an Agile methodology, such as Scrum or Kanban, to manage change intake.

Additionally, customers and stakeholders need visibility into the status of changes. Issue tracking systems like Jira or Azure DevOps can go a long way towards fulfilling these requirements.

Another key element of change management is defining acceptance criteria. The acceptance criteria determine when a change is considered complete. Common criteria include:

All automated tests pass
Change is in production and no Sev 1 or Sev 2 errors have been generated
Customers and stakeholders have tested the change and confirmed that it fulfills their requirements

A solid DevOps automation process compliments the change management process. By incorporating automated quality control elements such as test runs and monitoring alerts, your team can assess the basic health of changes without extensive manual testing.

Involve security experts

DevOps automation doesn't come without challenges. One of the most challenging elements is security.

Giving engineers the power to push changes more frequently to production is, overall, a positive development. However, engineers are often more focused on the task at hand than on ensuring the solution they're creating is secure.

The urge to ship quickly can encourage teams to make fatal mistakes, such as not securely storing credentials. Such seemingly small mistakes can have fatal consequences. Take for example Uber, which saw hackers steal the data of 57 million customers. The root cause? A set of credentials checked into GitHub.

Additionally, the tools used in DevOps automation present security challenges themselves. CI/CD tools like Jenkins and AWS CodeBuild require privileged access to sensitive systems. This makes them ripe attack targets for hackers.

This is why many DevOps experts urge teams to adopt a so-called "DevSecOps" model. In DevSecOps, security begins when the project begins and is baked into the same DevOps automation framework used to propagate changes to production. Your team can employ automated tools to enforce security policies, detect potential intrusions, and unearth known vulnerabilities in code and third-party libraries.

How TinyStacks can support your DevOps automation strategy

If you're new to DevOps automation, you likely read this article and thought to yourself, "That sounds like a lot of work." And indeed, it is. Establishing a solid DevOps automation process takes time and testing. And the work is never-ending. Your team will always find ways it can or should improve its DevOps pipeline.

At TinyStacks, we realized several years ago that the undifferentiated heavy lifting that DevOps requires was a high hurdle for many companies. So we decided to lend a helping hand.

TinyStacks is a DevOps pipeline solution that already includes cloud infrastructure creation, out of the box security, monitoring, approval gates, and more. All your team has to provide is its application code and TinyStacks does the rest. You can get started with DevOps automation without hiring a DevOps expert. Think of it as DevOps automation - automated!

Want to learn more? Contact us today for a demonstration and discussion on how TinyStacks can kickstart your DevOps automation strategy!