Agile development and retrospective coherence
Last week, Jurgen Appelo agitated quite a few agilists with his blog entry The Decline and Fall of Agilists (great post by the way – keep on writing good stuff, Jurgen! 
. He sees more and more people preaching agile and doing agile rigidly, proclaiming for instance that you can’t do Scrum without the eXtreme Programming (XP) practices.
He describes moving to agile development from the perspective of complex adaptive systems and game theory. Agility is about moving software projects from either an ordered context or a chaotic context into a complex context. Depending on where an organization comes from, different approaches work, e.g. Scrum with or without the XP practices.
This fits with my experiences. I see more and more agilists focusing on agile according to The Book or saying it’s not agile if you don’t do the sacred practices. It is also fits with the stuff Willem and I have been writing and presenting about Jerry Weinberg’s Cultural Patterns and Dave Snowden’s Cynefin sensemaking framework applied to software projects.
Cynefin sensemaking framework
Cynefin describes four domains, as shown in the picture below (the green arrows represent the agile choreographies Jurgen describes): the simple and complicated domains are ordered, you can see cause and effect relations there, either simple or complicated but still knowable. This means you can devise methods and processes that will lead to desired outcomes in a bounded, predictable way.
In the chaotic domain, no cause effect relationships are perceivable. The main decision model is to act quickly, trying to reduce turbulence, sense immediately what the effects of your action are, and respond accordingly. You can try to impose order on the system or reduce turbulence a bit to move towards the complex domain.
In the complex domain, the relationship between cause and effect can only be seen in retrospect. Looking back, causes and effects seem clear and we can explain what has happened and why – retrospective coherence. If you wouldn’t know better, it looks just like you’re in an ordered domain. But only in retrospect! You can’t reliably predict what will happen and you can’t reliably determine up front which actions will lead to desired results.
Retrospective coherence trap
I see a lot of agilists stepping into the retrospective coherence trap. They assume, consciously or unconsciously, an ordered context, where following a process like Scrum, XP, Scrum+XP, … will lead predictably to desired results (i.e. working software & a happy customer or product owner).
Most software projects that make a difference exist however in a complex context, because of a complex organizational context, ever changing customer requirements, complex problem domain, and/or unknown and rapidly evolving technology (I’ve written about this before in the context of Weinberg’s cultural patterns).
If you’re not aware that your project is in the complex domain, you might be tempted to reuse the processes of a successful project for a next project, assuming it will be similar. It might work by accident, but more often than not the context changes, not radically but enough for the process not to work as expected. This can explain why many Scrum and agile transitions still fail.
The retrospective coherence trap is also a risk when doing retrospectives if you focus mainly on what worked well without realizing things only make perfect sense when you’re looking back. In the complex domain, success stories are only retrospectively coherent and not a recipe for success.
Summarizing
- Software projects that make a difference usually are in the complex domain.
- Managing in the complex domain requires an approach that is fundamentally different from managing in an ordered domain. In the latter, it makes sense to follow known methods leading to project success or steer a project based on your (mental) models of how the project works (e.g. doing agile by the book). In complex space, you need to do numerous small interventions, experiments, short iterations, do small probes, sense what happens, and respond accordingly – the agile principle of apply, inspect, adapt.
- We often assume that we’re in ordered context, even if we’re in a complex domain.
- We step into the retrospective coherence trap: what happens makes sense only retrospectively, making it look like we’re in an ordered context. The context will however change in unknowable ways, so doing what worked yesterday will probably not give the expected results – although when you look back, it all makes perfect sense…
I do not really care whether you should do Scrum by the book, do all the 15.7 practices, get a high rating on an agile checklist, or even whether what I do is officially ‘Agile’. I strive to find what works for myself, for my teams, and for the organizations I work with. It all depends (the main lesson I learned at university), or context is king as Jurgen states it, and anything goes (from Against Method, required reading for agilists, but that’s another blog entry) I feel a theme emerging for the next Agile Open Europe conference ;-)
A few recommended links
- Cognitive Edge website with more information about Cynefin and related work
- Systems thinking
- Jurgen Appelo’s blog about understanding development and management
February 9th, 2009 at 11:39 pm
Nice post Marc!
Even though I strongly believe in the XP practices I do think that Jurgen has a good point, though I don’t agree with all his arguments or presuppositions. And, I think you bring his point even more home for me with what you call retrospective coherence. All the hoopla in the Agile community now about “doing Agile correctly” reminds me of Dale’s excellent post on the Null Process (http://cwd.dhemery.com/2003/09/the_null_process/).
Another point you make came as a bit of a surprise to me. About how focusing on what went well in retrospectives is not a recipe for success. Well, nothing is a recipe for success in our complex environment, as CAS theory tells us, but does this mean you focus more on dealing with the negatives? Am I on the wrong track learning about Appreciative Inquiry and Solutions Focus?
February 10th, 2009 at 8:08 pm
Thanks Daði,
I have little experience with AI and Solutions Focus, so I can’t say much about that. I know that Dave Snowden emphasizes that learning from failures and failure stories is much more effective than learning from success stories. He’s also quite critical of approaches like AI. On the other hand, I think it’s good to learn about multiple approaches realizing that there’s no best way or set of best practices. Having more perspectives can be useful when moving in the complex domain.
I’m still percolating about (agile) retrospectives as they are often performed and taught. I encounter teams that do retrospectives like following a kind of recipe – do the steps of the retrospectives, note stuff that works, keep doing that, change what doesn’t work, basically assuming an ordered context.
In my opinion, retrospectives in the complex domain work differently – the most important thing is then getting the stories out, sharing there, and having the right conversations. Retrospective structure, facilitation, program, planning, etc. are only a means (i.e. they work as boundaries, attractors, etc.) to let the conversations happen.
My ideas about retrospectives have recently been heavily influenced by the book “Changing Conversations in Organizations: A Complexity Approach to Change” by Patricia Shaw (highly recommended). I’m still processing it and intend to write more blog entries about retrospectives and changing conversations.
February 10th, 2009 at 9:52 pm
Thanks for the thoughtful reply. I’ll order the book you recommended next time we order books. I currently have “The Solutions Focus” waiting for me on my shelf, so it’ll be interesting to compare and contrast when I get through both.
Look forward to your blog entries about retrospectives and changing conversations
February 11th, 2009 at 10:04 am
Hi Marc,
Good to be reminded of this. Here in the UK we’ve been lucky enough to benefit from David Snowden’s presence. A number of folks in the agile community here and elsewhere – Joseph Pelrine, Steve Freeman, Rachel Davies amongst them – were Cynefin-accredited early on, and have been using these ideas effectively in practice and teaching for a number of years.
It’s important to remember, though, that this is just one model of sensemaking or situational awareness – there are many many others. Sometimes they simply recast things in different ways – you can take your pick, for example, between probe/sense/respond and Scrum’s empirical process control; sometimes they come at things from enough of a different angle to add a new dimension to your understanding (like the work of Ralph Stacey or Karl Weick).
As technologists we’re regularly beguiled by models (particularly when they use ego-boosting geek words like “chaos” and “complexity”
).
Agreed, too, that retrospective-by-recipe is a surefire way to guarantee that a team won’t get value out of the practice. It’s the most fragile of the agile practices, it depends more than any on a level of maturity and self-awareness in both coaches and teams that I see all to rarely.
March 17th, 2009 at 7:33 am
Nice post indeed. I also presented on this ‘do it by the book’ behaviour and Agile conformism. I look at it from a (product) adoption point of view. A set of Agile principles are great for early adopters and innovators, but the large masses desperately need very pragmatic and defined ways of doing things. Scrum + XP fits that need. This also means that, like any non-evolving product, it will die of relatively quickly, much like RUP did (did it die?). Mostly because there will be new ways to ‘make your entire company competitive ™’ but also because few people will do Scrum + XP correctly. The large masses also have a tendency to use a watered down version of things (as apposed to ‘tuned’) and then blame the method for their failure. Perhaps they need some systems thinking to solve that problem