My notes from the DevOps Handbook

by Gene Kim, Jez Humble, Patrick Debois, John Willis

Swarm and solve problems to build new knowledge

Mobilize whoever is required to solve the problem.

Goal of swarming:

• contain problems before they have a change to spread
• diagnose and treat so that it cannot recur

Toyota Andon cord: pull and fire an alarm whenever something goes wrong so that everyone can see. If the problem cannot be solved within a time frame (for example 50 seconds) the whole production line is halted, so that entire organization is mobilized to help solve the problem.

Instead of working around the problem and fixing when we have time, we swarm to fix it immediately.

Swarming is necessary because:

• it prevents problem from progressing downstream
• cost to repair grows exponentially and accumulates technical debt
• it prevents work center from starting new work that will introduce new errors

Swarming enables learning because it prevents loss of critical information due to fading memories.

Swarming of ever smaller problems discovered ever earlier in the lifecycle that we can deflect problems before a catastrophe occurs.

Create a culture that is encouraged to pull the Andon cord.

Keep pushing quality closer to the source

Pushing decision making further away from where the work is performed:

• lowers the quality of the decisions
• increases cycle time
• decreases feedback strength (cause and effect)

Examples of ineffective quality controls:

• requiring another team to complete error prone task that can be automated and self-serviced
• requiring approval from busy people that are distant from the work and knowledge
• low quality documentation that becomes quickly obsolete
• large batches of work approved by special committees

Instead we need everyone in our value stream to find and fix the problems in their area of control and push the quality and safety responsibilities and decision making to where the work is performed.

It is impossible for a developer to learn anything when someone yells at them for something they broke six months ago - that's why we need to provide feedback to everyone as quickly as possible, in minutes, not months.

Enable optimizing for downstream work centers

Types of customers in Lean:

• external - the one who pays for the product
• internal - who receives the work to process in the next stage of the value stream - this one is the most important to us

We should optimize for downstream work centers by designing for operations, where operational non-functional requirements are prioritized as highly as user features.