Lowering the cost of change is a frequently cited benefit of adopting agile development and extreme programming, in particular. You'll frequently see graphs drawn that show an exponentially growing cost curve as the cost of change for a waterfall project and graphs that show a level or even down sloping (after an initial ramp up cost) cost curve for XP projects.

I personally have been in involved in projects where the down sloping cost curve was actually achieved, so I know it is possible. I also know it is rare; on other projects a flat cost curve was all that was achieved. Both are good outcomes, but what is the cause for the differences in these projects? How does a project actually achieve low costs of change?

However, before exploring how to lower the cost of change, defining the cost of change is necessary. I frequently see discussions about the cost of change that center around the time or effort required to implement a story. While not wrong, I think this perspective is too narrow and short sighted. This definition is not affected by a team that is "going in circles". But who cares if they can go in circles really fast?

Mary Poppendieck has discussed a Maturity Model of Software Development. (She has a presentation on this and an article published in Software Development magazine. http://poppendieck.com/maturity.htm) The basic idea is that what matters is how quickly and reliably can an organization take a user request and deliver it in the software. This is the cost of change I wish to focus on: the amount of time elapsed and resources spent on taking a user feature and having it present in the software application.

Now, returning to analyzing the cost of change itself, there are two considerations: what contributes to the cost of change and how does agile development help lower it.

Contributors to the cost of change:

• time to assess and prioritize a user request
• overhead for specifying a story
• "churn" in stories
• number of changes to code base to implement story
• cost of release

How is feedback handled? How many people are part of the decision? The longer it takes for a sales person to communicate a user requirement to product management, have it prioritized and then worked on by engineering, the hight the cost of change. Similarly, the number of people (or worse yet, committees) that must approve or deny the request, the higher the cost of change.

The time and number of people it takes to specify a story and complete set of acceptance tests affect the cost of change.

The higher the number of times a feature area is revisited, be it because of bugs or revisions due to what was delivered wasn't quite what user's wanted, the higher the cost of change.

The time it takes to assess and discover the necessary code changes and to implement them impacts the cost of change.

What is the overhead to a release? Is there a manual QA cycle? How long does it take? Documentation done at the end?

How can Agile lower the Cost of Change?

The primary mechanism for lowering the cost of change is bridging the gaps between the user, product owner, developer and application. (For more on this, please see this blog post.)As the gaps are closed, the application tends to be in alignment with the user requirements.

Practices such as an onsite customer and acceptance tests help develop a shorthand of communication between the product owner and the developers. Thus, the overhead of specifying stories tends to go down and existing acceptance tests and infrastructure serve as templates and extension bases for new acceptance tests.

Iterations provide many benefits. They allow opportunities for injecting feature requests, shortening the time between a user's initial request and when she sees it in the product. Iterations, combined with engaging a cross functional team in each iteration, keeps activities such as documentation and exploratory testing in step with development, shortening release cycles.

Finally, the engineering centric nature of XP encourages the application, and more importantly the underlying code base, to become ever closer to expressing the user's intentions and needs. It is very gratifying as a developer when your code base almost anticipates new user features.

I think software teams should actively try to measure, monitor and lower the cost of change. The complete "cost chain", from sales to development, needs to be included and managed. Agile practices, such as XP, will provide a lot of guidance for improving the elements closest to the code, while applying general agile principles, such as cross functional teams, iterations, high communication, emphasis on feedback, can help the other aspects not directly involved with development. An organization that can successfully manage and lower the cost of change will achieve significant competitive advantages such as lower cost and time to market. Organizations that can't manage the cost of change will have trouble competing at all.

I think in a software project there are varying notions of reality regarding the actual application being developed. They are:

• actual needs of the market or user base
• product owner's vision of the application
• developer understand of the goals and requirements of the application
• code base and actual executable application

This is represented in the following diagram:

The gap between these perspectives is normal, particularly at the start of a project. However, I think it is difficult for a project to succeed if the gap is not being closed; there needs to be alignment and overlap among the different viewpoints.

One of the motivations of agile software development, at least in my opinion, is to help close these gaps. The high bandwidth of communication, intense collaboration, emphasis on feedback and the importance of acceptance tests are techniques to communicate the assumptions, needs and current state of each different constituent. With each iteration, the gap should be bridged a little more.

This raises an interesting question: How can you measure or monitor the shrinking (or worse the expanding) of the gap?

I don't think you can measure this directly. However, there are a variety of indicators that can help evaluate how well you are closing the gap.

Bugs are probably the best indicator. A bug represents a mismatch in expectations, understanding or communication. Bugs are reported when a calculation is incorrect as well as when a user doesn't like the number clicks it takes to perform an operation. In both cases, there are one or gaps between what the user needs and what the application is providing; perhaps the product owner specified a requirement that made the user experience awkward or a developer missed a test case. If your bug rates aren't going down it is an indicator that the gaps are not shrinking. I suggest assessing the cause for the constant (or growing) bug rate by evaluating which gap is the widest. Understanding where the communication / comprehension breakdown exists is the most important step in resolving it.

There are other indicators as well, such as the amount of churn in stories (requirements) and the overhead of specifying stories. If you are revisiting the same areas of the application frequently, whether the reason is user dissatisfaction or defects, there are gaps between the user and the product owner or between the developer, user and code base.

There are ultimately two reasons for closing the gap: delivering a product that users like (and will buy) and lowering the cost of change. One reason is a huge component of succeeding today, the other allows you to succeed tomorrow. When the gaps are small and there is alignment between the user, the application and all those in between, you will probably have a winning project.

My blog has not seen a post in a long, long time... My last post even promised that I would post more.

Excuses aside, I do sincerely plan a lot more posts in the coming future. As I noted in an earlier post, I have taken a position at one of my clients, Nominum. I've been an employee a bit over a year and it has been a very interesting experience.

I've been ranting for a while that XP is an engineering practice and that there is more to successful software projects than what agile development offers. I've been critical about the silence (in my opinion) of the agile community and literature on the missing elements that actually have the largest impact on a successful project. I had theories (some at least partially verified in my experience) about the interdependencies between the various elements of the "value chain" in a software project.

My experience at Nominum has smacked me in the face with all the things I've ranted about. Its confirmed many ideas, but also posed many challenges. It is still a work in progress, but I revisted and revised many of my ideas regarding XP and agile software development. I plan to post about them in the near future.