My notes from the DevOps Handbook

by Gene Kim, Jez Humble, Patrick Debois, John Willis

67. Protecting the Deployment Pipeline

Integrate security and compliance into change approval processes

If we have constructed our deployment pipeline correctly so that deployments are low risk, the majority of our changes won't need to go through a manual change approval process, because we will have placed our reliance on controls such as automated testing and proactive production monitoring.

Change processes defined in ITIL:

• Standard changes: These are lower-risk changes that follow an established and approved process, The change proposer does not require approval before deploying the change, and change deployments can be completely automated and should be logged so there is traceability.
• Normal changes: These are higher-risk changes that require review or approval from the agreed upon change authority.
• Urgent changes: These are emergency and, consequently, potentially high-risk changes that must be put into production immediately. These changes often require senior management approval, but allow documentation to be performed after the fact. A key goal of DevOps practices is to streamline our normal change process such that it is also suitable for emergency changes.

Re-categorize the majority of our lower risk changes as standard changes

One way to support an assertion that our changes are low risk is to show a history of changes over a significant time period (e.g., months or quarters) and provide a complete list of production issues during that same period. If we can show high change success rates and low MTTR, we can assert that we have a control environment that is effectively preventing deployment errors, as well as prove that we can effectively and quickly detect and correct any resulting problems.

Even when our changes are categorized as standard changes, they still need to be visual and recorded in our change management systems. Ideally, deployments will be performed automatically by our configuration management and deployment pipeline tools and the results will be automatically recorded.

We may automatically link these change request records to specific items in our work planning tools. This can be accomplished by including ticket numbers from planning tools in the comments associated with version control check ins.