Today I got a question in email about Scrum and Lean, fallout from the recent workshop I did with Mary Poppendieck at MIT. Is Scrum related to Lean? Are we finding out new things about Scrum or Lean? Are customers asking for Lean?
After thinking about the early days of Scrum I think the root of both Scrum and Lean is complex adaptive systems theory. When we created Scrum we did not talk about Lean, we talked about complex adaptive systems. I think Scrum and Lean are complementary implementations of ways to deal with physical reality where things are often not linear, not simple, and not predictable.
Of course, the name of Scrum comes from Professors Takeuchi and Nonaka looking at teams at Toyota, Honda, 3M in the United States, and other "lean" companies. Small cross functional teams with a facilitative team leader delivering production prototypes in short intervals reminded them of the game of Rugby and the Scrummage formation. Thus Scrum is "lean" by definition but Takeuchi and Nonaka never mention the word "lean." They view it as a western concept that tries to describe assembly line practices at Toyota which are constantly changing and are not the core value of the Toyota Way. 95% of the profit of Toyota is delivered by the Chief Engineers who are the ones doing Scrum.
The first Scrum implemented all the practices we know as Scrum today plus all of what later became known as XP practices plus some secret sauce that creates hyperproductive teams.
Pair programming was used regularly with Jeff McKenna, the senior consultant, who spent half a day with each developer when he was at Easel one week a month. After one of these sessions developers were heard complaining that 75% of their code base was thrown out in a refactoring frenzy. The 25% remaining did everthing more cleanly and elegantly.
Continuous integration was the norm. It still astounds me that developers think they can program production systems without this. (They can but they suck!)
The concept of an "Ensemble" was the key to component based design. We had sophisticated component based testing strategies based on silicon chip design.
We had experienced engineers on the QA team automating tests at the component level. This was feature level testing. It was a forerunner to the smoke test that runs on every build in my later companies.
We deployed builds for use at the end of each monthly Sprint. Each build had sufficient functionality for Easel consultants to use it on real projects.
But most important, we captured the "punctuated equilibrium" effect, something I have yet to see another Scrum team do. It is one of the secrets of hyperproductive teams. Before a developer chooses a task in the Scrum meeting, the entire team discusses with the developer whether that task will be the fastest path to an observable end-use feature. This was a sophisticated implementation of "self-organization" and "test first" development.
After the first Scrum started at Easel in 1993, I began talking about the process in the internet newsgroups in 1994, particularly comp.object, and by 1995 we had a well documented thread of my discussions on Scrum in comp.object, interlaced with comments from Bob Martin, Kent Beck, and others.
So the origins of Scrum are well documented and can be researched in Google groups by searching in the archives. My original Scrum web page started in 1994 and was widely viewed on the web throughout 1995 and updated regularly until the Agile Manifesto meeting in 2001.
During 1995, I invited Ken Schwaber to take a look at the first Scrum team which led to collaborating on the first formal paper on Scrum presented in the fall of that year at the OOPSLA. We did not discuss Lean at all but did discuss complex adaptive systems.
Mike Beedle was influenced by the online description of Scrum, implemented the process in his own company, and led the effort to drive Scrum through the Pattern Languages of Programming Design conferences. This made Scrum the first (and only) formal organizational pattern that describes a complete Agile process. Recent work by Jim Coplien shows that Scrum is deceptively simply while compressing a complex array of organizational patterns in his book, "Organizational Patterns in Agile Software Development."
Looking at "scrum.txt" you will see that Scrum derives directly from best practices in new product development documented by Takeuchi and Nonaka in their 1986 HBR paper. The leading example in the paper was Honda (Toyota was not so well know at the time). Takeuchi does not specifically address Lean as we know it today.
In 1995, I commented in the comp.object newsgroup:
Usability studies ... have convinced us that only one paper in the literature describes the rapid application development environment achieved with Object Studio (the combination of Enfin Smalltalk and Synchronicity Business Object Modeling and Persistent Object Mapping).
Takeuchi, Hirotaka and Nonaka, Ikujiro. The New Product Development Game. Harvard Business Review, Jan/Feb 1986, pp. 137-146.
Takeuchi and Nonaka describe the Scrum methodology (a Rugby term) which we use internally. It is significantly different than anything you have seen because there is no project management schedule, only a committed delivery time for a release. The Japanese auto and electronic companies are using it to clean our clocks in the global economy. Did you notice that U.S. auto companies had a banner year and still lost market share in 1994?
Small teams are given objectives to meet in a fixed time frame. They take the existing software (you never build anything from scratch) and small projects (2 days to 2 weeks per person) are assigned as SynchSteps. SynchSteps are dynamic pulses generated against the existing code structure that causes mutations (as in a biologic organism).
A project team is typically 3 developers, 3 QA people, 3 doc people, and one or two users. They meet daily and all agree on steps completed and next steps. For large projects, small teams of this size build components and a SCRUM of SCRUMs meets less frequently to work out interfaces between components. Developers must be outnumbered on the team by QA and documentation people or they generate too much code too fast (malignant functionality).
The code evolves like a biological system via punctuated equilibrium. You can read about this process as modeled by Denny Hillis on a Connection Machine in the book "Artificial Life." When enough mutations occur in multiple parts of the organism, the system shifts to a higher plateau of functionality. We say at this time that a "package" is emitted. This will be a new piece of the software system we are building. For example, we are adding Use Cases to the Synchronicity product at this time and when the package appears, an objective for the release cycle is met.
A release cycle, typically six months, is built of packages which are loosely defined as objectives at the start of the release cycle.
Management must release control of the team and let it function as a self-organizing entity that grows a system like a plant. We have found that we get dramatically more functionality in less time and a much cooler product with this approach. This approach rationalizes the comments that Fred Brooks made in 1987 in his paper "There is No Silver Bullet". There, he stated that the only way to speed up development was to grow a prototype like a plant.
We track the process by the velocity of SynchStep (Sprint Backlog item) assignments versus the velocity of SynchStep completion. This gives estimate of package time completion, kind of like launching a rocket, watching its trajectory, and predicting impact point. We can adjust the impact point by lowing the arc (less functionality) or adding rocket fuel (more resources) to hit the time frame committed to management.
We monitor progress regularly via demonstrations. Nothing counts with management except delivered code that works. Packages are regularly shipped out as alpha components that people can drop into the current release of Synchronicity. Synchronicity dynamically changes, menus and all, so that people can try out the new functionality and see if they like it.
With SCRUM schedules are obsolete (developers love it) and delivery dates never slip (management loves it). If this sounds too good to be true, I suppose it is, but management will agree there is more functionality and less slippage with this process than anything they have ever seen. Our users are asking us to lengthen the product delivery schedule because we are delivering too many new releases with major upgrades
too fast. (This still doesn't stop them from complaining about not getting everything on their wish list.)
We find that the paridigm shift required for optimum utilization of the SCRUM process is significant. The object-oriented paridigm requires a shift in thinking, but Scrum requires a shift in the organization and the way people work and relate to one another.
------------------------------------------
Scrum was strongly influenced by complex adaptive systems theory - the work of Peter Senge at MIT, Christopher Langdon at the Sante Fe Institute, and many others. Also by Goldratt's constraint theory and by Rodney Brooks subsumption architecture, the basis of the company iRobot which I helped to get started shortly before Scrum began. Starting at the bottom of my original Scrum web page I commented:
"SCRUM is based on complexity theory and artificial life experiments at Thinking Machines using highly parallel simulation of systems evolution. It induces the phenomenon known as "punctuated equilibrium" seen in the evolution of biological species."
The reason the first Scrum team called Sprint Backlog items SyncSteps was that developers executed the Sprint Backlog in a carefully chosen order - that order which produced the fastest path to appearance of the next feature from the users point of view. Just as proper ordering of the Product Backlog optimizes revenue, proper ordering of the Sprint Backlog optimizes production of value. It accelerates software evolution and can produce effects seen in biological systems.
The "punctuated equilibrium" effect is achieved at Toyota using set-based concurrent engineering. As an example, Toyota does not build one radiator for a new car. They build six, and wait to the last possible moment to choose the best radiator to deploy in production. This is similar to competition between biological species in an ecosystem and the species that adapts best to the environment wins. The Scrum community has yet to implement set based concurrent engineering strategies that were used by the first Scrum team.
At Google I was asked why no Scrum teams implement this set-based conconcurrent engineering strategy. This forced me to thing hard about the problem. The answer appears to be that Scrum teams (1) have no clear and consistent component architecture, (2) the team members do not have an overall view of the architecture and state of each component, and (3) the teams have no formal method for automated testing at the component level.
However, test first development drives the software creation process in the right direction as early testing shows immediately what does not work and allows rapid deployment of alternative strategies. The key is evolution of software like a biological system based on competing implementation constructs and survival of the fittest.
Test first development is described in my paper on Scrum in a CMMI Level 5 company where accepted tests for features are described first, the feature is implemented in priority order and tested immediately. Systematic Software engineering showed this doubled velocity and cut defects in half. Test Driven Development is writing test code before writing functionality and is useful in the right hands. This tends to focus at the unit test level and helps with refactoring. As Ron Jeffries pointed out in our joint MIT workshop, he then let's the code do the talking and the architecture emerges. However, many developers cannot hear the code do the talking and can generate a mess where there is more test code than production code and more bugs in the test code than in the application. I can see the design form in Ron's mind as he works on the code and it is clear he is not the average programmer. He sees the design space alternatives unfolding as he codes and has the good judgement to take the right path into a flexible and maintainable architecture. It reminds me why Jim Copien asserts that "Architect Codes" is a highly important organizational pattern that is best practice in leading companies. (Jim also asserts TDD is broken for reasons alluded to here.)
A thoughtful analysis of the relationship between complex adaptive systems and Lean will show that both Scrum and Lean are instantiations of complex adaptive systems theory. A subset of complex adaptive systems theory is artificial life, a term invented by Christopher Langton at the Sante Fe Institute in the 1980's. His seminal paper showing that organisms similated on a computer evolve faster as they approach chaotic regions strongly influence Scrum. In Scrum, we introduce chaos into the development process and then use an empirical control harness to inspect and adapt a rapidly involving system.
In trying to describe Scrum at JAOO in 2005, I used Lean to show why Scrum works. Scrum is a way to implement lean in building software. In fact, it has the advantage that if you follow it closely and implement well, you will be doing Lean as articulated by Mary and Tom Poppendieck sithout even understanding Lean. It is hard for people to understand complex systems theory so it is difficult to use that as a motivator. It is hard for people to understand Lean as well, but Toyota has become such a dominant force in industry that fear, uncertainty, and doubt wakes people up from their sleep.
In 1995, I worked with Hubert Smits and Jean Tabaka at Rally to develop a course for people who want to take their Scrum implementation to the next level all based on Lean as the easiest way to articulate what you need to focus on to optimize Scrum. Ken Schwaber argues that by simply focusing on the Scrum framework and the impediment list you will achieve this. This is absolutely correct. Lean is just an extra teaching tool.
There is also the paper on Scrum and CMMI Level 5 on the implementation of Scrum by Systematic Software Engineering, an implementation entirely driven by Lean. In Denmark, it is national policy to introduce lean practices in all industries. As a result, it is easy to sell Scrum as a way to implement Lean in software.
Scrum is an inspect and adapt framework which is extremely simple by design to allow the average developer at Ford Motor company to get started in a couple of days. However, it is hard to implement. Less than 10% of the Scrum teams worldwide can pass the Nokia test, primarily because they cannot deliver potentially shippable (fully tested) software at the end of a Sprint. Talking to people about complex adaptive systems doesn't seem to help them as well as talking to them about Toyota to show them how their implementation of Scrum is broken, how it is riddled with disruptions in flow (muri), stressed everywhere (mura), and waste (mudah). Using Toyota as an example of a successful company who systematically elimiates muri, mura, and mudah shows them how they might best go about fixing their process.
So Lean is helpful as a teaching tool because it derives from the basic laws of the universe which are articulated at a theoretical level by complex adaptive systems research. Scrum derives from the same fundamental principles. Lean has practical ways to deal with process mapping, the cost of delays, queuing problems, reset times, and so forth. These are all useful in dealing with constraints, global system throughput, and other factors which were a primary concern for Scrum in dealing with the complex adaptive system called software development.
Lean derives from Edward Deming and the work of others teaching the Japanese that local optimization caused by traditional management by objectives was self-destructive. Deming, after consulting with the leadership in many American corporations, concluded that management is totally broken in the U.S. causing incalculable and inconceivable damage (such as the destruction of the auto industry). The global optimization promoted by Deming and his statistical quality control practices are rooted in complex adaptive systems theory, constraint theory being a specialized area that focuses on global optimization. Professor Senge's work on systems theory at MIT is the most readable resource for these early influences Scrum.
So the bottom line is that:
1. Scrum is not a direct descendant of Lean. It derives from complex adaptive systems with an indirect linkage to Lean through Takeuchi and Nonaka.
2. Management is interested in Lean, partcularly in Europe. Building on that interest and any knowledge they have about Toyota makes it easier to describe why Scrum works.
3. Lean is a good teaching tool to show Scrum teams why their implementations are broken. For example, if you do not have fully tested code and cannot create potentially shippable code in a Sprint, you have 100% work in progress going into the next Sprint.
Excess work in progress is viewed as a horrible and intolerable blunder in a Lean operation, yet software development teams and management seem to have difficulty understanding why this work in progress will double your defects and cut your velocity in half. And even when they understand it they often lack the political will to fix it. Teaching Kaizen mind and stopping the line is useful medicine for this disfunction.
English 
French 
Spanish 
English