The value of automation
The compilation of applications and services comes in all shapes and sizes. It may seem straightforward that code just becomes a packaged artifact, but for many scenarios, builds can have a number of complex steps with many dependencies. The steps involved in creating and testing an artifact may be manual, automated, or a combination of both to form a build pipeline. The following figure demonstrates an example build pipeline with a set of activities critical to the building of an application:
Figure 1.1 – Example build pipeline
Running these builds manually could potentially lead to careless mistakes with an outcome that may not be consistent or repeatable. When code is being built, it is very important to document even the smallest changes that were made to the build that resulted in something working as opposed to not working. This is where the use of a source code management (SCM) system, such as Git (https://git-scm.com), becomes critical in our overall pipeline. If a change was actually the result of a build step changed locally, not being able to repeat this can result in frustration and productivity loss.
This is especially relevant from the perspective of handing off work to a colleague. Having to understand and tweak a set of manual steps in a build would not be a good use of that colleague’s time, when they could instead be focused on the code at hand. The time of each member of an organization is valuable and it’s best to allow that individual to focus on being productive. This could be during a production outage, where time is best spent trying to fix the root cause of the outage rather than analyzing how to actually build the code. Depending on the impact of the outage, every second could have a monetary impact on the organization. In cases of a simple development handoff to a production outage, automation of a build would be very beneficial to the situation.
Imagine if a developer could solely focus on code development, rather than analyzing manual steps or difficult-to-execute builds. An organization might have automation in place, but it must be seamless for the developer in order to maximize productivity. This can be in the form of the following:
- Coding standards
- Boilerplate code
- Blueprints on how to use the pipeline
The preceding reference points could also apply to both what is considered the automation of the inner loop and the outer loop of software development. The inner loop of development typically consists of local development by a developer. Once code is completed in the inner loop, a merge request is created for addition to an integration branch. Once merged in an integration branch, this is where the typical build pipeline starts: the outer loop. Providing a starting point in the form of standards itself may not be automation; however, it could be baked into the configuration files. It may just be a starting point, a foundation that can also provide a level of flexibility for the developer to apply specific preferences.
Figure 1.2 – Example inner and outer loops
The ecosystem of tools and integration that has been built around Git has helped drive the importance of version controlling not only source code but also configurations that define a build pipeline. GitOps (https://www.weave.works/blog/the-history-of-gitops) primarily focuses on infrastructure as code (IaC), ensuring that a runtime environment represents configurations declaratively stored in Git. The common use of Git tooling across developer and operation teams reduces the amount of friction for onboarding, which makes GitOps also critical for end-to-end automation.
Automation helped reduce end-to-end deployment times for this organization: https://cloud.google.com/customers/reeport/.
Once automation is streamlined, the team that owns the pipeline is able to aggregate metrics in order to determine areas for improvement at scale. This becomes a more achievable task, compared to when builds are executed manually by each developer. As mentioned earlier, pipelines in an organization could also include manual steps. These metrics could identify patterns where manual steps could possibly be automated. Reducing manual steps would increase the efficiency of a pipeline while also reducing the potential human errors that can occur. There may be situations where manual steps aren’t automatable, but identification is key so that it can be considered in the future or to allow for teams to focus on other steps that can be improved.
This can reduce developers’ frustration and improve overall productivity across teams, which can benefit the organization in the following ways:
- Delivering features faster
- Reducing the amount of time it takes to resolve issues
- Allowing teams to focus on other business-critical activities
- Feedback for continuous improvement of the pipeline
The value of automation can help an organization in many aspects. While metrics can be manually gathered, they can be most effective when aggregated in an automated pipeline. Decisions can be made to determine the most effective changes to a build pipeline. The metrics gathered from frameworks in place, such as GitOps, can also help feed into improving the end-to-end pipeline, not just the automation of source code compilation. Continuous improvement becomes more achievable when an organization can use metrics for data-driven decisions.