Last week, Jurgen Appelo agitated quite a few agilists with his blog entry The Decline and Fall of Agilists (great post by the way – keep on writing good stuff, Jurgen! . He sees more and more people preaching agile and doing agile rigidly, proclaiming for instance that you can’t do Scrum without the eXtreme Programming (XP) practices.
He describes moving to agile development from the perspective of complex adaptive systems and game theory. Agility is about moving software projects from either an ordered context or a chaotic context into a complex context. Depending on where an organization comes from, different approaches work, e.g. Scrum with or without the XP practices.
This fits with my experiences. I see more and more agilists focusing on agile according to The Book or saying it’s not agile if you don’t do the sacred practices. It is also fits with the stuff Willem and I have been writing and presenting about Jerry Weinberg’s Cultural Patterns and Dave Snowden’s Cynefin sensemaking framework applied to software projects.
Cynefin sensemaking framework
Cynefin describes four domains, as shown in the picture below (the green arrows represent the agile choreographies Jurgen describes): the simple and complicated domains are ordered, you can see cause and effect relations there, either simple or complicated but still knowable. This means you can devise methods and processes that will lead to desired outcomes in a bounded, predictable way.
In the chaotic domain, no cause effect relationships are perceivable. The main decision model is to act quickly, trying to reduce turbulence, sense immediately what the effects of your action are, and respond accordingly. You can try to impose order on the system or reduce turbulence a bit to move towards the complex domain.
In the complex domain, the relationship between cause and effect can only be seen in retrospect. Looking back, causes and effects seem clear and we can explain what has happened and why – retrospective coherence. If you wouldn’t know better, it looks just like you’re in an ordered domain. But only in retrospect! You can’t reliably predict what will happen and you can’t reliably determine up front which actions will lead to desired results.
Retrospective coherence trap
I see a lot of agilists stepping into the retrospective coherence trap. They assume, consciously or unconsciously, an ordered context, where following a process like Scrum, XP, Scrum+XP, … will lead predictably to desired results (i.e. working software & a happy customer or product owner).
Most software projects that make a difference exist however in a complex context, because of a complex organizational context, ever changing customer requirements, complex problem domain, and/or unknown and rapidly evolving technology (I’ve written about this before in the context of Weinberg’s cultural patterns).
If you’re not aware that your project is in the complex domain, you might be tempted to reuse the processes of a successful project for a next project, assuming it will be similar. It might work by accident, but more often than not the context changes, not radically but enough for the process not to work as expected. This can explain why many Scrum and agile transitions still fail.
The retrospective coherence trap is also a risk when doing retrospectives if you focus mainly on what worked well without realizing things only make perfect sense when you’re looking back. In the complex domain, success stories are only retrospectively coherent and not a recipe for success.
- Software projects that make a difference usually are in the complex domain.
- Managing in the complex domain requires an approach that is fundamentally different from managing in an ordered domain. In the latter, it makes sense to follow known methods leading to project success or steer a project based on your (mental) models of how the project works (e.g. doing agile by the book). In complex space, you need to do numerous small interventions, experiments, short iterations, do small probes, sense what happens, and respond accordingly – the agile principle of apply, inspect, adapt.
- We often assume that we’re in ordered context, even if we’re in a complex domain.
- We step into the retrospective coherence trap: what happens makes sense only retrospectively, making it look like we’re in an ordered context. The context will however change in unknowable ways, so doing what worked yesterday will probably not give the expected results – although when you look back, it all makes perfect sense…
I do not really care whether you should do Scrum by the book, do all the 15.7 practices, get a high rating on an agile checklist, or even whether what I do is officially ‘Agile’. I strive to find what works for myself, for my teams, and for the organizations I work with. It all depends (the main lesson I learned at university), or context is king as Jurgen states it, and anything goes (from Against Method, required reading for agilists, but that’s another blog entry) I feel a theme emerging for the next Agile Open Europe conference ;-)
A few recommended links
- Cognitive Edge website with more information about Cynefin and related work
- Systems thinking
- Jurgen Appelo’s blog about understanding development and management