Dreamfeed

Transitioning to agile means going from a Variable or Routine culture to a Steering culture. Indicators for a Steering culture are the ideas of empirical process control, embrace change, and apply-inspect-adapt.

The agile community provides all kinds of process models that are in fact mental models of software development: e.g. Extreme Programming, Scrum, Lean Software Development, Crystal Clear, the stories people tell, external consultants’ pet peeves. These are a good source for routines to choose from.

Practices like daily standup meetings, short iterations, frequent releases, iteration retrospectives, test driven development, continuous integration, big visible charts, stop the production line mentality all help improve the visibility and stability of the software development system.

An agile transition has a number of failure modes. I will briefly describe three of them.

Agile backlash

Introducing visibility can be scary. I have seen and heard about multiple XP and Scrum initiatives that got killed because the sudden visibility of problems and (lack of) progress was threatening to middle management. Instead of becoming a Steering culture, it remains Variable or Routine. I’ve met for example a manager who waved away any agile practice as soon as it didn’t fit with his own agenda, saying that you shouldn’t be dogmatic with agile…

To reduce the backlash problem, ensure higher management support for your agile initiative.

Agile by the book

Now that agile has crossed the chasm, I see more and more organizations that do agile ‘by the book’, basically going through the motions without a good understanding of why they are doing it (saying e.g. that you should stick to the rules). This is ok if it’s a temporary phase in the transition process. When people start understanding the ‘why’, the organization can transform to an effective Steering culture.

If a team gets stuck in this phase, it basically settles on a Routine culture. It will go awry sooner or later and the people will blame the agile ‘method’ for the problems.

An experienced agile coach can help you through this phase.

Agile getting into a rut

Steering is based on finding deviations between observations and objectives. This is the basis for improvement. If everything is running smoothly and going so well that there are no obstacles any more, there’s a risk of complacency. Peope forget why they are doing the things they do and the system slowly becomes a Routine culture.

To avoid this failure mode, you can train yourself and your team to become conscious of this process and learn to introduce changes and experiments just for the sake of change. Learn how to keep the software development system awake by jiggling it. It’s a first step towards an Anticipating culture.

Would you like coaching and guidance in transitioning effectively to a Steering culture? Would you like to learn how to jiggle your agile system and how to keep it alive and kicking? We can help you - feel free to contact us.

• Ronald Doelen has written a nice blog entry about his CITCON Europe experiences
• James Shore wrote about an energized work experience; the story is an instance of the systems diagrams I published earlier this week.
• I discovered a good article on the Satir change model by Steven M. Smith.

• Willem and I have been working on working on a proposal for a new website for a not-for-profit organisation supporting a community of practice. This has resulted in a nice systems diagram about the dynamics of attracting traffic to the website and the associated costs and benefits.

To establish an effective Steering culture, you need:

• a mental model of how the software development system works
• process visibility
• process stability

Mental models

You need an implicit or explicit mental model of how your actions affect the system and its outputs. Routine managers tend to use simple, linear models, relying on prediction by extrapolation. This leaves little room for variation and unforeseen changes. Managing in a Steering environment means you apply more sophisticated mental models, taking into account nonlinearity and cause-effect relations separated in space and time (systems thinking).

A typical Routine manager for instance assumes the amount of work done is directly proportional to the number of hours put in. So when the project is getting behind, he mandates lots of overtime. His mental model is illustrated by the causal loop diagram below.

The diagram shows three variables that you can measure or observe: the number of hours per week, the amount of finished work, and the amount of work left to do. If the number of hours per week increases, the amount of work that gets finished increases as well (indicated by the causality arrow and the +); if the amount of finished work increases, the amount of work left to do decreases (the inverse relationship is indicated by the -). This is a balancing loop (indicated by the scale symbol), because at a given moment the work left to do decreases rapidly enough to be finished before the deadline.

What happens in reality is that things speed up at first, but after a few 80 hour weeks, the exhausted developers create more and more defects and the project completely runs out of control. The diagram below shows the dynamics. After a delay (indicated by ||), fatigue will take its toll, resulting in more defects and more work for the developers.

This creates a self-reinforcing loop (indicating by the snowball), pushing the system towards collapse - resulting in things like burn out, people getting sick, people leaving.

A Steering manager knows that overtime can work for a short time only. After that, people will become stressed and tired and will make more and more mistakes that add up to the work left to do. She also knows that the effect of work to do on the hours per week is a management decision: she can decide not to increase hours, but instead make sure the team gets enough rest.

Note that a Steering manager will usually not get into this situation; she will detect early on that the project risks getting late, when there is ample time to change the plan, get new people up to speed, or renegotiate with the customer.

Visibility

If you can’t observe or measure what’s happening, you’re not able to know if things deviate from the plan and you have nothing to base your actions on. As an example, you can observe how a culture handles failures and defects.

In a Variable culture, where customers and developers work closely together, the customer reports a defect directly to the developer, who fixes it right away. This makes it hard to get insight in things like the amount of defects reported and solved or the amount of new defects introduced after fixing a defect.

A Routine culture doesn’t regard a defect as information, but as something to assign blame for. As a result, defects are moved around like hot potatoes, ignored, blamed on the user, classified as ‘minor’ (i.e. it won’t get fixed - learn to live with it), or relabeled as a change request. Managers buy fancy bug tracking tools, but these are not used effectively. Because of the blaming associated with defects, making defect handling more visible will run into a lot of resistance.

Scrum and Extreme Programming provide simple and effective ways of increasing project visibility, e.g. through velocity, burndown charts, story walls, or kanban boards. So these process models can help you transition from a Variable or Routine culture to a Steering culture.

Stability - If the software development processes are not stable, you don’t know what the effects of your actions are. If e.g. your project velocity varies wildly over many iterations, your burndown chart becomes meaningless, and you can’t promise your customer anything.

Introducing automated testing at different levels (unit, integration, functional, acceptance) helps to stabilize your projects.

Links

• George Dinwiddie, It’s all about feedback
• Peter M. Senge, The Fifth Discipline: The Art & Practice of The Learning Organization, about systems thinking, mental models, and organizational learning.
• Donella H. Meadows, Places to Intervene in a System
• Linda Booth Sweeney & Dennis Meadows, The Systems Thinking Playbook

Would you like to explore systems thinking and mental models as a tool for better understanding and better decision making? Would you like to get a grip on your projects? We can help you - feel free to contact us.

Last weekend, I participated in CITCON Europe 2007 in Brussels. CITCON stands for Continuous Integration and Testing Conference. It is part of a worldwide series of short Open Space conferences. The session results can be found on the CITCON wiki.

I liked the conference: good atmosphere, interesting people, met a few new people and a number of the usual suspects at agile conferences. The Open Space format proved again to be a good process for high energy and peer to peer exchange of experiences. It was just a bit short - one evening with opening and agenda creation plus one full day of sessions.

Being an Open Space fan, I was also interested in how Paul Julius and Jeffrey Fredrick facilitated the process. They did a good job at organizing self-organization ;-). For a lot of participants this was their first Open Space event and they liked it. There also was self organizing wall, with a growing number of sticky notes, about the next conference location, people looking for work, work looking for people… - great idea, I’d like to do something like that for the next Agile Open.

I liked the Helping teams to change behavior session in particular, hosted by Jeffrey. It was about ideas, tools, and techniques for bringing change in teams. He mentioned models like Crossing the Chasm by Geoffrey Moore and the Satir Model of Change. The stuff was new to most of the participants in the room.

Links

• Open Space Technology
• Agile Open conferences
• Satir Model of Change (pdf)

Would you like to learn more about how the Satir tools can help your teams change effectively? We offer workshops and coaching, feel free to contact us.

This is the next installment in a series of blog posts on Jerry Weinberg’s cultural patterns of software development organizations.

A Steering culture focuses on improving what you develop, through steering the software development system based on feedback about its outputs. Steering applies the concept of feedback control to software development management.

Oblivious, Variable and Routine cultures are based on lack of trust and power. Steering, Anticipating, and Congruent cultures on the contrary are trust-based: managers and customers trust developers to deliver a good job and create quality software, developers trust managers to remove impediments and to provide guidance instead of micromanagement, everyone is part of one team. I find a trust-based environment much more enjoyable to work in. It is also more efficient because it reduces the amount of communication and time spent on contract negotation.

Steering software development means observing the outputs (which includes software, bugs, happy team members, burned out programmers, etc.), comparing the observations to a plan, and taking action if a deviation from the plan is found. The actions are supposed to reduce the difference between what is observed and what is desired. In the field of organizational learning this steering process is called single loop learning. See also the illustration below (based on the diagrams in Quality Software Management vol. 1 by Jerry Weinberg).

The controller can be management, but this is not necessary. It includes anyone who influences the system, like a project manager, a Scrum master, an agile coach, or a technical lead.

A Steering culture introduces different ways of feedback and uses it appropriately. Testing is an important form of feedback. In Routine organizations, testing is a phase at the end, which in practice results in uncontrollable chaos and quality problems. In a Steering organization, testing is an measurement process that provides information for decision making and interventions. Testing is something that you can add to different processes: unit testing for rapid feedback on the impact of the code you’ve just changed, continuous integration for feedback on integration problems, automated acceptance tests for feedback on what’s done.

An example from an agile project:

After the review of the current iteration, you determine the current velocity, the amount of work remaining, and the number of open defects (observations).

You notice that you won’t be able to finish the work planned for the release next month unless you remove 5 stories from the plan (comparing to the plan).

You discuss this with the customer, who juggles priorities and cuts scope for this release (action).

Other examples of actions are: adjust the plan, add or drop requirements, add, remove, or train people, give a pep talk to the team, celebrate team success, and do nothing.

When everything is going well, Routine and Steering organizations look the same. When problems arise, a Steering organization has a repertoire of routines and is able to adapt instead of panicking. In a Steering culture people learn to pay attention to subtle signals and seemingly insignificant data, because big, runaway problems often start as small, manageable deviations - act early, act small. Daily standup meetings are an effective tool for surfacing issues quickly. Also pay attention to the emotions in your team, these are an indicator of other troubles. A Steering manager can use weekly informal one-on-one meetings with everyone as a tool to gauge emotions.

In a Routine culture, processes are defined but often not understood. This is why people drop them as soon as things run awry. In a Steering culture, processes are not always completely defined, but they are understood. You know why you do the things you do. As a result, you will be more inclined to stick to the processes in hard times. A Routine culture regards processes as explicitly definable and executable by interchangeable programming resources developers.

A Steering culture acknowledges that parts of the processes can be explicitly defined (explicit knowledge) but that there is also tacit knowledge involved - the know-how in the heads of the people, which cannot be written down easily.

Links

• Chris Argyris & Donald A. Schön, Organizational Learning II: Theory, Method, and Practice, about single and double loop learning
• Ikujiro Nonaka & Hirotaka Takeuchi, The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation, about explicit knowledge vs. tacit knowledge
• George Dinwiddie, It’s all about feedback

Do you wish to transition to a trust-based, steering culture? Would you like to learn how to steer your projects effectively? We can help you - feel free to contact us.

Routine - we follow our routines (except when we panic)

A Routine culture assumes that there is a optimal way of doing things and that we can determine this best way. This assumption can hold if the organizational context, the problem domain, and the technology are well-known and predictable enough.

A Routine culture might work if you mainly do small, simple, similar projects. I think a Variable culture fits this context better however, being more efficient and less risky - a group of skilled craftsmen who can oversee the work and who will do whatever they think is best to get results.

If your organization specializes e.g. in building accounting software and you have built over a dozen accounting systems with the same technology, the same DBMS, the same programming environment, a Routine culture could work. You should ask yourself however why you are doing the same thing over and over again - you’re better off making a product or service out of it and selling this product or service. Alternatively, you could make a framework or automate it (e.g. an accounting systems generator or a 4GL for accounting systems). This will give a much better return on investment.

So some IT projects are in a known, predictable context, but most of them aren’t. Usually the problem domain is not as predictable as you wish. The business context changes continuously. You have to discover requirements along the way. By deploying prototypes and early versions of software, you influence the users’ perception and understanding of problem and solution, making the whole system unpredictable.

Even if the problem domain and technology are well-known and predictable, you still have to cope with the complexity of people systems when you roll out a system. If you have one or two users, you might be able to oversee it…if you have to roll out your system to ten different offices and hundreds of users, you’re definitely in a complex space…

IT projects that deliver a strategic advantage for the business are by definition not in a known, predictable context - if it was known and predictable, there wouldn’t be any strategic advantage, only something you need to do to keep up with the competition.

And even if everything is currently known and predictable, it’s not going to stay that way:

• You tread into a new and unknown problem domain, thinking something like: nah, client relationship management is probably just like accounting.
• Someone else creates an accounting service that costs ¼ of what you charge, undermining your company’s reason for existence.
• Your programming language or 4GL isn’t supported any more and you have to migrate to .NET, a technology that is completely new to you.

This makes Routine processes break down. People start to panic. In practice, the software development system becomes more and more like a Variable culture, where most people just do whatever they think should be done in order to get over with it. This is not something that is discussed or admitted (don’t you dare mention it!). Officially, you’re still following your routines. In practice, you’re doing something completely different.

An example: in a regular waterfall project, a test phase is planned at the end of the project, followed by a few days of rework, followed by deployment to production. What happens in practice, is: the test phase uncovers a lot of defects all at once, developers work 70+ hour weeks to fix everything; they start cutting corners in code quality and unit testing as a response to the management pressure to finish before the release date; ultimately, even a few untested fixes slip into production.

How can you recognize a Routine culture going awry?

• Lack of trust; focus on role definitions, contracts, service level agreements instead of collaboration.
• People say “that is his/her responsibility, not mine!”
• Lots of blaming (and placating) - managers look for scapegoats and reprimand developers for not following the process - Who’s fault is this? Heads must roll!; developers blame managers, but not openly.
• Developers complain about all the routine meetings, but still attend them, silently nodding and yawning.
• People try to stick more and more to the routines, adding extra procedures and rules; but this makes things even worse.

So what can you do when this happens? You could let the routines fade away over time and proceed in the chaotic way things are already happening, i.e. transitioning to a Variable culture.

Alternatively, you can try transitioning to a Steering culture, but this is not for the faint of heart: you have to build trust and make the undiscussable difference between what is should happen and what actually happens discussable. One of the next installments will be about transitioning from a Variable or Routine culture to a Steering culture.

In the mean time, don’t forget to check out Nynke’s blog entry on scenario planning as an Anticipating culture practice.

Links

• Most IT Jobs Are Like Deodorant - about strategic, value adding IT work vs. essential but not strategic IT work.
• The new dynamics of strategy: Sense-making in a complex and complicated world - in depth article about the Cynefin model, about known, the knowable, complex, and chaotic spaces.
• Gerald M. Weinberg, Quality Software Management vol. I: Systems Thinking