Scrum, Agile, Complexity and Ants

I started re-reading Agile Project Management with Scrum by Ken Schwaber a few days ago, after not having looked at it in far too long. I was struck by the similarities between Schwaber’s view on the complexities of software development and the concepts of Complex Systems Theory I recently read while helping someone do research for a paper. I have to think that Schwaber and the other founders of Scrum were at least partly inspired by it.

Complex Systems Theory is, not surprisingly, difficult to describe in a nutshell. A complex system is a system composed of actors that interact with and influence each other in many different ways. It is essentially a network with nearly infinite relationships between the actors. Natural ecosystems are good examples, where millions of animals, plants, bacteria and fungus interact and affect each other in uncountable ways.

Complexity Science studies how this variety of actors and interactions can result in “spontaneous self-organization.” The system organizes itself, adapts, and evolves without anyone directing or planning the organization. The adaptation is a result of actors constantly adapting to each other. The international stock markets dropping in reaction to a news report is an example. No one person held a meeting of all the world’s stock brokers and said “SELL SELL SELL!” Some read the news and adjusted their portfolios. This lead to others adjusting their portfolios and so on. Another example from nature is a community of ants changing to a new food source. The queen ant didn’t send out royal directives. One ant found a new food source and interacted with some other ants and then your picnic basket was overrun.

The study of complex systems has also been applied to business, in particular to management and the engineering and manufacturing of products, as a way to encourage innovative thinking and rapid responses to change. Traditional management relies on the idea that the business environment is predictable, and so business strategy and product development can be planned months ahead and instructions can be passed down the hierarchy. But anyone trying to manage a requirements document in a software development project will tell you a different story. The modern business world is unpredictable, fast moving, and incredibly complex. Complexity science tells us that if given the right environment, the people in an organization and their relationships can spontaneously organize and adapt to change much faster and more successfully than a hierarchically controlled system. Thus Agile works because it is empirical and adaptive, not a defined process. The team experiments, learns and validates continually rather than adhering to a restrictive and sometimes inappropriate process.

Re-reading Schwaber’s book reminded me that with the complexity of software development, success requires the self-organization of the team. I often get lost in the ‘process’ of scrum, the sprints and scrum meetings and backlogs and reviews, forgetting that scrum is almost an anti-process. The ‘process’ of scrum is simply creating as little process to dictate behavior as possible, and instead ensuring the optimum environment for the team to self-organize. Scrum is about removing impediments and distractions and enhancing communication and collaboration. The sprints and reviews are not intended to tell the ants how to get to the picnic basket. They’re there to enable the the team to find a faster and better way to the picnic basket, even if someone keeps moving it.