Systems thinking with diagrams of effects

October 13th, 2009

Rachel has written a nice blog entry on how to create a diagram of effects. Diagram of Effects (or causal loop diagram) is a systems thinking technique, based on the work by Jerry Weinberg and Peter Senge. I started using them about 8 years ago, to make sense of my experiences with agile software development.

Over the years the technique has proven useful, not only in coaching and workshops, but in all kinds of situations:

  • As a sensemaking technique – to make sense of what’s happening in your team or at your client, so that you can take more appropriate, effective action. I started doing systems thinking together with a number of other agile coaches around 2002. It helped us a lot to understand better what agile is all about. We realized among other things that it’s not about my Scrum being better or worse than your XP, but it’s about doing whatever works for your situation.
  • To illustrate the dynamics of software development, e.g. technical debt (pdf) or the dynamics of test driven development and pair programming.
  • As a means to do root cause analysis in retrospectives – diagrams of effects facilitate team problem solving processes (similar to what David Draper is doing with current reality trees, another systems thinking technique).
  • To support sales conversations – we are using diagrams of effects more and more in sales conversations, e.g. clarifying how a website delivers value for a nonprofit organization or explaining the effects of portfolio and risk management.

picture  by Willem van den Ende

Like Rachel, we create diagrams of effects collaboratively, in small groups, focusing on the conversations and not so much on the diagrams. It is also possible to use the diagrams in a more mechanistic way, creating a precise model of the system and even using it to run simulations. This approach is suitable for knowable problems and situations, not for complex systems like teams and organizations.

Some of the effects we have seen in practice, are:

  • Diagrams of effects work as a medium that enables deep conversationsthe conversations that change organizations. The value lies in the process of creating, refactoring, and discussing a diagram collaboratively, less in the resulting diagram (although the diagram can be useful as an illustration, to retell the story).
  • They help surface your assumptions. For each causal relation you draw, ask yourself: what assumptions is this relation based on? How can you test these assumptions?
  • They shift focus towards the whole system and make clear how everyone/everything plays together, instead of blaming individuals. You’ll see the underlying patterns of behaviour through the incidents. It can help you build your own situated theory of the team/organization dynamics.

So, will diagrams of effects solve all your problems? No, but they are a easy to learn, simple, useful technique that will enrich your repertoire of skills and techniques. As a result you will have more options to choose from and you’re better able to do what fits the situation.

Would you like to know more? Jerry Weinberg’s Quality Software Management: Systems Thinking and Peter Senge’s The Fifth Discipline are a good start. Alternatively, take a look at systemsthinking.net or contact me.

<a href=”http://www.amazon.com/gp/product/0415249147?ie=UTF8&tag=piecemealgrow-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0415249147″>Changing Conversations in Organizations: A Complexity Approach to Change (Complexity and Emergence Inorganisations, 6)</a><img src=”http://www.assoc-amazon.com/e/ir?t=piecemealgrow-20&l=as2&o=1&a=0415249147″ width=”1″ height=”1″ border=”0″ alt=”" style=”border:none !important; margin:0px !important;” />

Conference appearences in October

October 6th, 2009

This month, I will run two workshops (together with Willem van den Ende) at the following conferences:

Story Testing at the Scandinavian Agile Conference
15-16 October in Helsinki (Finland)

As a developer, I want to know story testing frameworks and practice one of them, so that I can test stories using the examples the product owner writes together with me.

As a product owner, I want to understand the process of testing stories and practice specifying acceptance criteria using example scenarios, so that I can help developers develop what I want.

This workshop focuses on behaviour driven development and writing automated tests for user stories based on examples, with story testing frameworks like Cucumber or JBehave. Learn about specification by example, tools for testing user stories; practice hands on writing and improving story tests using example scenarios.

Mapping Product Line Value Streams at the
Practical Product Lines conference
20-21 October in Amsterdam (NL)

The goal of this workshop is to apply and evaluate value stream mapping as a tool for understanding business value creation in a complete software development organisation from requirements to delivery. We wil use value stream mapping to analyse existing processes or product lines brought in by the participants.

We use simple tools like index cards to prevent discussion about drawing techniques and enable all attendees to participate in creating a value stream map.

Would you like to register for the Practical Product Lines conference at a 10% discount? Just let me know.

Working with User Story Mapping

September 23rd, 2009

We are using more and more user story mapping, in our courses and with clients. User Story Mapping is a relatively new agile product planning practice, with its roots in user centered design. It’s being popularized by Jeff Patton.

User story mapping offers an alternative for traditional agile planning approaches like the Scrum product backlog. Instead of a simple list, stories are laid out as a two dimensional map. The map provides both a high level overview of the system under development and of the value it adds to the users (the horizontal axis), and a way to organize detailed stories into releases according to importance, priority, etc. (the vertical axis). The map shows how every user story fits in the full scope.

Releases are defined by creating horizontal slices of user stories, each slice is a release. For the first release, it is recommended to build a minimal set of user stories covering all user goals, so that you build a minimal but complete system to validate functionality and architecture early.

An important contribution of user story mapping is its focus on the users, the user goals, and the (system independent) activities/processes that the system should support. This helps the team to focus on why they are building the software – business value instead of feature details.

Story mapping was a natural fit for us, after working with agile for a number of  years. It augments existing agile processes and solves a number of issues we have run into.

We love small user stories

We came to prefer small user stories that can be build in 1 or 2 days. Small stories have all kinds of advantages. They are easier to estimate, you deliver working software in small steps (working software daily), you get early feedback and a continuous feeling of accomplishment. We don’t need to do extensive task planning, because making a quick list of tasks just in time on an index card is sufficient.

Limitations of traditional agile planning approaches

We ran into a number of limitations of small stories. It becomes increasingly more difficult to understand the system as a whole through all the small stories. Some of our customers had trouble seeing the forest for the trees. We were also lacking a way to manage coherent sets of dependent stories. These start off as epics or themes, but once they are split up, it becomes hard to keep track of what belongs to what and how far each theme or epic has been done.

Where to Go  by Jeff Hitchcock

Where to Go by Jeff Hitchcock (found with PhotoSuggest)

We also ran into limitations of linear backlogs for managing and prioritizing user stories. With lots of small stories and a lack of a good overview, it becomes difficult to prioritize. You spend more and more time on rearranging and grooming the backlog grooming instead of delivering working, valuable software.

Release planning – determining what value to deliver when – is a complex problem with multiple dimensions. Reducing it to a one dimensional backlog can work for simple systems, but for most systems we work with, essential details are lost.

The traditional way of agile planning supports working incrementally well. It provides little support for iterative development (i.e. building a story by fleshing it out in a number of deliveries). The risk here is that customers expect a user story to be finished in one go, while it might need two or more iterations to be fleshed out well. Lack of support of iterative development also bears the risk of customers requesting complete, bloated features (because it should be done right the first time), where simpler features would be sufficient (but the customer cannot know that until he/she sees the working features).

Last but not least, we frequently encounter developers and customers discussing primarily in terms of software features. Teams get lost in the solution details without a thorough understanding of the actual problems the software will solve. It’s not only developers failing to speak the customer’s language! We have also encountered quite a number of customers and users talking databases, fields, screens, and other solution aspects, instead of focusing on what they actually want to achieve.

User story mapping to the rescue

User story mapping focuses on the different users of the system and their goals. To achieve their goals, users perform activities, consisting of tasks. These are all user-centric – focusing on what the user does, not on features of the software. Features should support the user in doing tasks and achieving goals.

Instead of a linear backlog, you create a map of user’s activities and tasks. This defines the system at a high level and gives a complete overview of the scope. You put activities and tasks along the horizontal axis. The map then tells the story of the system: the activities describe the bird’s eye view of the scope, the tasks tell how a user actually uses the system. Note that activities and tasks are closely related to business processes. Adapt the terms to your own context.

You define releases based on tasks. You iteratively build the system, fleshing out the different tasks by defining detailed user stories (which are software oriented). The vertical axis of your story map indicates importance: higher up means more important.

You define releases by drawing horizontal slices and moving stories up and down between slices. For the first release you define a walking skeleton – the bare bones of the system, minimal functionality that covers all activities. You defer splitting out the task level stories into concrete, detailed stories until the last responsible moment.

Detailed stories will become ‘done’; activities en tasks are never ‘done’ as a story; you can always put in more features, make it better.

There are no hard and fast rules for story mapping. There are a number of useful guidelines, but no strict boundaries of what activities, tasks, and stories are. The boundary between user oriented stories and software oriented stories is not strictly defined either.

Benefits

We see a number of benefits of using user story mapping.

  • It helps to keep an overview of the whole system and makes visible what value the system adds, what goals and activities or processes it supports, and how all the small, detailed user stories fit in.
  • It facilitates release planning and enables a better conversation about defining delivering value early and managing risks; building a walking skeleton can give early feedback on market and technical risks.
  • By introducing the users perspective and the rationale behind the system, User story mapping shifts the dialogue to the actual business value of the software. We only build features when the story map shows a rationale for that feature. A story map helps focus on the essential parts of the system. We have also used story maps to make teams aware that you don’t need to build all those features. A subset is often sufficient to help users achieve their goals.
  • By focusing on user goals, activities, and tasks first instead of software features, you keep more options open regarding ways of realizing / supporting those goals and activities. Manual workarounds can be a good alternative more often than you’d think (btw I prefer to call it sapient processes rather than manual workarounds)
  • It augments rather than replaces practices from e.g. Scrum or eXtreme Programming and it plays well with other techniques like Dimensional Planning and agile business process analysis.

Would you like to know more? Jeff Patton has written very good material on story mapping. There’s however only so much theory, the beef is in doing it. We have developed a simulation called The New New NEW! Product Development Game to experience release planning with user story mapping. We can also guide you in applying user story mapping in your projects. Let me know if you’re interested.

Agile Open Holland 2009

July 6th, 2009

We’re running another Agile Open conference this year! It’s scheduled for September 10 & 11, in Baarn (right in the middle of The Netherlands).  Agile Open is a two day, full Open Space conference, interactive, peer-to-peer, intense, with lots of fun & learning.

This year we’re organizing it together with Agile Holland. Many thanks to VxCompany for helping us out by providing the venue. 

The conference theme is:

My method is bigger than yours…or is it?

A development method is nothing more than a named cloud – a set of practices, principles and values labeled under a common name. Whether you realize it or not, each of these named clouds have fuzzy boundaries and overlaps with others. Examples of named clouds you may be familiar with include Scrum, eXtreme Programming, Test Driven Development, DSDM, Lean, Refactoring and more recently Kanban, User Story Mapping, and Dimensional Planning.


Word cloud of agile, Scrum, Kanban & XP created with Wordle.

This year, we have space for 80 participants. See the announcements on the Agile Open website (English) and the Agile Holland website (Dutch) for details.

Hope to see you in Baarn in September!

Pending comments

July 6th, 2009

I’ve been away from blogging for some time and apparently I missed a number of comment notifications :-(  All the pending comments have been approved now (or spammed). Sorry to all the commenters who did not see their comments appear until today.

I promise to get back to blogging and check pending comments more often…

Agile development and retrospective coherence

February 9th, 2009

Last week, Jurgen Appelo agitated quite a few agilists with his blog entry The Decline and Fall of Agilists (great post by the way – keep on writing good stuff, Jurgen! ;-) .  He sees more and more people preaching agile and doing agile rigidly, proclaiming for instance that you can’t do Scrum without the eXtreme Programming (XP) practices.

He describes moving to agile development from the perspective of complex adaptive systems and game theory. Agility is about moving software projects from either an ordered context or a chaotic context into a complex context. Depending on where an organization comes from, different approaches work, e.g. Scrum with or without the XP practices. 

This fits with my experiences. I see more and more agilists focusing on agile according to The Book or saying it’s not agile if you don’t do the sacred practices. It is also fits with the stuff Willem and I have been writing and presenting about Jerry Weinberg’s Cultural Patterns and Dave Snowden’s Cynefin sensemaking framework applied to software projects.

Cynefin sensemaking framework

Cynefin describes four domains, as shown in the picture below (the green arrows represent the agile choreographies Jurgen describes): the simple and complicated domains are ordered, you can see cause and effect relations there, either simple or complicated but still knowable. This means you can devise methods and processes that will lead to desired outcomes in a bounded, predictable way. 

In the chaotic domain, no cause effect relationships are perceivable. The main decision model is to act quickly, trying to reduce turbulence, sense immediately what the effects of your action are, and respond accordingly. You can try to impose order on the system or reduce turbulence a bit to move towards the complex domain.

In the complex domain, the relationship between cause and effect can only be seen in retrospect. Looking back, causes and effects seem clear and we can explain what has happened and why – retrospective coherence. If you wouldn’t know better, it looks just like you’re in an ordered domain. But only in retrospect! You can’t reliably predict what will happen and you can’t reliably determine up front which actions will lead to desired results.

Retrospective coherence trap

I see a lot of agilists stepping into the retrospective coherence trap. They assume, consciously or unconsciously, an ordered context, where following a process like Scrum, XP, Scrum+XP, … will lead predictably to desired results (i.e. working software & a happy customer or product owner).

Most software projects that make a difference exist however in a complex context, because of a complex organizational context, ever changing customer requirements, complex problem domain, and/or unknown and rapidly evolving technology (I’ve written about this before in the context of Weinberg’s cultural patterns). 

If you’re not aware that your project is in the complex domain, you might be tempted to reuse the processes of a successful project for a next project, assuming it will be similar. It might work by accident, but more often than not the context changes, not radically but enough for the process not to work as expected. This can explain why many Scrum and agile transitions still fail.

The retrospective coherence trap is also a risk when doing retrospectives if you focus mainly on what worked well without realizing things only make perfect sense when you’re looking back. In the complex domain, success stories are only retrospectively coherent and not a recipe for success.

Summarizing

  • Software projects that make a difference usually are in the complex domain.
  • Managing in the complex domain requires an approach that is fundamentally different from managing in an ordered domain. In the latter, it makes sense to follow known methods leading to project success or steer a project based on your (mental) models of how the project works (e.g. doing agile by the book). In complex space, you need to do numerous small interventions, experiments, short iterations, do small probes, sense what happens, and respond accordingly – the agile principle of apply, inspect, adapt.
  • We often assume that we’re in ordered context, even if we’re in a complex domain. 
  • We step into the retrospective coherence trap: what happens makes sense only retrospectively, making it look like we’re in an ordered context. The context will however change in unknowable ways, so doing what worked yesterday will probably not give the expected results – although when you look back, it all makes perfect sense…

I do not really care whether you should do Scrum by the book, do all the 15.7 practices, get a high rating on an agile checklist, or even whether what I do is  officially ‘Agile’. I strive to find what works for myself, for my teams, and for the organizations I work with. It all depends (the main lesson I learned at university), or context is king as Jurgen states it, and anything goes (from Against Method, required reading for agilists, but that’s another blog entry) I feel a theme emerging for the next Agile Open Europe conference ;-)

A few recommended links