Lean Principles Make Software Development Anti-Fragile

[part of a series]

The Nature of the Software/IT Domain

Software development projects, and IT projects in general, are complex cognitive endeavors. The more people, teams and technologies involved, the more complex it gets. Further, it is often a very creative process – while many elements (especially those at small scale) are repeatable, more often completely new artifacts must be invented.

When projects start, the executives funding the effort legitimately want to know, how long will this take? That is, how much is it going to cost?

Over the last 50 years there have been many approaches to providing these estimates. Some work better than others, but they all have one thing in common: They will be wrong.

Completion Estimates in Extremistan

Completion estimates are always wrong.  But the interesting thing is they always (nearly always) underestimate the time to complete. Why is that? It is the rare, memorable occasion when a project finishes early.

The fundamental mistake people make is thinking software development lives in Mediocristan, where outcomes are confined to nice clean Gaussian Bell Curves. But it does not.

Software development lives in Extremistan, with fat tails: the “lateness” of any effort is unique, and can be orders of magnitude greater than anything you’ve see so far, or that anyone else has seen.

And these fat tails are nearly always concave – worse and worse.

What do you do when you work in Extremistan? You need protocols for precaution.

Fortunately, Lean Principles provide the basis for those protocols.

Number One: Limit your exposure

Start with the Lean Principle of Small Batch Size. “Batch” can mean the number of items, the size of the work item, the detail and depth of fidelity (e.g., prototypes, MVP's), or the length of time the work proceeds before being “checked on”. All of these are appropriate for software development.

This means that you quickly "surface" the work and see if it is on track, providing the value you expected, conforming to the standards you set, etc.

If (when) things are off-track, they are not off for very long, because only a "small batch" of wrongness has be created.

The ultimate expression of this is the Lean Principle of One Piece Flow, where you only make a single (small) item at a time, with the whole team involved, and then check how you did.

Number Two: Hypothesis and Test – Learn from your experiments

Second is the Lean Principle of Continuous Improvement – the Plan, Do, Check, Act cycle. Underneath is the basic driver of all progress in the last 400 years: The Scientific Method.

Start with the Lean Principle of Visual Management: Before you begin your small batch, articulate everything you know. Make visible (physical if possible) all elements of the work, all artifacts, thoughts, plans, diagrams, code, etc. This is your hypothesis: we hypothesize that running this (visualized) system will lead to an anticipated result.

When your small batch finishes, conduct a thorough forensic analysis what happened. What did you learn? What can be done better? Make this analysis specific and actionable. Collect data (e.g., cycle time) and review against all previous experiments.

You want to learn the “physics” of your domain, as much as possible. Make explicit what you know, why you think you know it, and what it means.

There will still be unknowns - that's the nature of risky, fat tailed worlds. But keep moving forward with skeptical empiricism: use everything you have learned so far, keep learning - and expect to change your mind when new information is obtained.

Number Three: Make That Learning Cycle Immediate

Lean encourages people to continually monitor the progress of the work, and raise any problem immediately. This is the Lean concept of the Andon cord, and the Lean Principle of Jidoka: Problems are Treasures. It means stop work at the slightest hint of any problem, confusion, loss of value, quality concern or waste. 

The response of the entire organization is gratitude to the person for caring enough about the customer to first notice, and then act. The stoppage is treated as a learning opportunity for all people, systems, and artifacts involved.

This in turn drives the “batch size” down to that same level of immediacy. Because the Lean response is to use that call to “stop the train” to improve, right there and then.

Number Four: Eliminate Waste – Signal to Noise

The fourth Lean Principle to being anti-fragile is to Eliminate Waste. There are 15 "kinds" of waste: the list of categories is a learning aid that helps sensitize people to the problem. It is a key part of value stream analysis.

The main value that eliminating waste brings to being anti-fragile is it removes "noise" from the system. This "noise" is unnecessary complexity. Each reduction in complexity will allow you to focus more on the smaller, local scale work. Which again helps limit your exposure to fat tail events.

Your Learning Organization's Ensemble of Projects becomes Anti-Fragile 

These practices (when implemented by competent leadership) create a learning organization that is the heart of being anti-fragile.

Why? Because anti-fragile organizations get better. They learn from each mistake, each problem, each setback, and do better next time. And that's what Learning Organizations built on Lean Principles do.

Any given project, any given moment in the progress of the work, is fragile, open to the unknown possibility of catastrophic failure.

But with Lean, the project ensemble, and the ensemble of projects underway, becomes anti-fragile.