Software Project Management: Adding Stakeholder Metrics to Agile Projects

Abstract

Agile methods need to include stakeholder metrics in order to ensure that projects focus better on the critical requirements, and that projects are better able to measure their achievements, and to adapt to feedback. This paper presents a short, simple defined process for evolutionary project management (Evo), and discusses its key features.

Introduction

In 2001, a British Computer Society Review paper indicated that only 13% of 1027 surveyed IT projects were 'successful' (Taylor 2001). In the same year, a Standish report indicated that although there has been some recent improvement, 23% of their surveyed projects were considered total failures and only 28% totally successful (that is, on time, within budget and with all the required functionality) (Johnson 2001: Extracts from Extreme Chaos 2001, a Standish Report). The US Department of Defense, a few years ago, estimated that about half its software projects failed (Personal Communication, Norm Brown, SPMN (Software Program Managers Network)/Navy). While these figures represent an improvement on the 50% reported for failed DoD projects when the Waterfall Method dominated (Jarzombek 1999), they are still of extreme concern. We must be doing something very wrong. What can senior management and IT project management do about this situation in practice?

Some people recommend complex development process standards such as CMM (Capability Maturity Model®), CMMI (Capability Maturity Model® Integration), SPICE (Software Process Improvement and Capability dEtermination) and their like. I am not convinced that these are 'good medicine' for even very large systems engineering projects, and certainly they are overly complex for most IT projects.

Other people recommend agile methods - these are closer to my heart - but maybe, for non-trivial projects they are currently 'too simple'?

Stakeholder Metrics

I believe agile methods would benefit if they included 'stakeholder metrics'. I have three main reasons for suggesting this:

• to focus on the critical requirements: All projects, even agile projects, need to identify all their stakeholders, and then identify and focus on the 'top few' critical stakeholder requirements.

• to measure progress: Critical requirements need to be quantified and measurable in practice. Quantified management is a necessary minimum to control all but the smallest upgrade efforts.

• to enable response to feedback: By responding to real experience and modifying plans accordingly, projects can make better progress. This is something that agile projects with their short cycles can especially utilize.

In this paper, I shall present a simple, updated 'agile', evolutionary project management process and explain the benefits of a more focused, quantified approach. I recommend the evolutionary project management process and policy shown in Figure 1.

A Simple Evolutionary Project Management Method

Tag: Quantified Simple Evo Project. Version: July 8, 2003. Owner: Tom@Gilb.com.

Status: Draft.

Project Process Description

1. Gather from all the key stakeholders the top few (5 to 20) most critical performance goals (including qualities and savings) that the project needs to deliver. Give each goal a reference name (a tag).

2. For each goal, define a scale of measurement and a 'final target' goal level. For example, Reliability: Scale: Mean Time Between Failure, Goal: >1 month.

3. Define approximately 4 budgets for your most limited resources (for example, time, people, money, and equipment).

4. Write up these plans for the goals and budgets (Try to ensure this is kept to only one page).

5. Negotiate with the key stakeholders to formally agree these goals and budgets.

6. Plan to deliver some benefit (that is, progress towards the goals) in weekly (or shorter) cycles (Evo steps).

7. Implement the project in Evo steps. Report to project sponsors after each Evo step (weekly, or shorter) with your best available estimates or measures, for each performance goal and each resource budget.

   • On a single page, summarize the progress to date towards achieving the goals and the costs incurred.

   • Based on numeric feedback, and stakeholder feedback; change whatever needs to be changed to reach the goals within the budgets.

8. When all goals are reached: "Claim success and move on" (Gerstner 2002). Free the remaining resources for more profitable ventures

Project Policy

1. The project manager, and the project, will be judged exclusively on the relationship of progress towards achieving the goals versus the amounts of the budgets used. The project team will do anything legal and ethical to deliver the goal levels within the budgets.

2. The team will be paid and rewarded for 'benefits delivered' in relation to cost.

3. The team will find their own work process, and their own design.

4. As experience dictates, the team will be free to suggest to the project sponsors (stakeholders) adjustments to the goals and budgets to 'more realistic levels.'

This simple project process and policy captures all the key features: you need read no more! However, in case any reader would like more detail, I will comment on the process and policy definition, statement by statement, in the remainder of this paper.

Project Process Description

1. Gather from all the key stakeholders the top few (5 to 20) most critical goals that the project needs to deliver.

   Projects need to learn to focus on all the stakeholders that arguably can affect the success or failure. The needs of all these stakeholders must be determined - by any useful methods - and converted into project requirements. By contrast, the typical agile model focuses on a user/customer 'in the next room'. Good enough if they were the only stakeholder, but disastrous for most real projects, where the critical stakeholders are more varied in type and number. Agile processes, due to this dangerously narrow requirements focus, risk outright failure, even if the one identified 'customer' gets all their needs fulfilled.

2. For each goal, define a scale of measurement and a 'final target' goal level. For example, Reliability: Scale: Mean Time Before Failure, Goal: >1 month.

   Using Evo, a project is initially defined in terms of clearly stated, quantified, critical objectives. Agile methods do not have any such quantification concept. The problem is that vague targets with no quantification and lacking in deadlines do not count as true goals: they are not measurable, and not testable ideas.

   Note that in Evo the requirements are not cast in concrete, even though they are extremely specific. During a project, the requirements can be changed and tuned based on practical experience, insights gained, external pressures, and feedback from each Evo step (See also point 4 under 'Project Policy').

3. Define approximately 4 budgets for your most limited resources (for example, time, people, money, and equipment).

   Conventional methods do set financial and staffing budgets, but usually at too macro a level. They do not seem to directly, and in detail, manage the array of limited resources we have. Admittedly there are some such mechanisms in place in agile methods, such as the incremental weekly (or so) development cycle (which handles time). However, the Evo method sets an explicit numeric budget for any useful set of limited resources, effort, calendar time, financial spend, or memory space.

4. Write up these plans for the goals and budgets (Ensure this is kept to only one page).

   All the key quantified performance targets and resource budgets should be presented simultaneously on a single overview page. Additional detail about them can, of course, be captured in additional notes, but not on this one 'focus' page.

5. Negotiate with the key stakeholders to formally agree these goals and budgets.

   Once the requirements, derived from the project's understanding of the stakeholder needs, are clearly articulated, we need to go back to the real stakeholders and check that they agree with our 'clear' (but potentially incorrect or outdated) interpretation.

   It is also certainly a wise precaution to check back later, during the project evolution, with the stakeholders, especially the specific stakeholders who will be impacted by the next Evo step:

   • as to how they feel about a particular choice of step content (that is, how they see the proposed design impacting performance and cost, and whether the original impact estimates are realistic in the real current implementation environment, and

   • to check for any new insights regarding the long term requirements.

6. Plan to deliver some benefit (that is, 'progress towards the goals') in weekly (or shorter) cycles (Evo steps).

   A weekly delivery cycle is adopted by agile methods; this is good. However, the notion of measurement each cycle, on multiple performance and resource requirements, is absent.

   Using Evo, the choice of the next Evo step is based on highest stakeholder value to cost ratios. It is not simply, "What shall we do next?" It is "What is most effective to do next - of highest value to the stakeholders with consideration of resources?"

   The agile methods' notion of agreeing with a user about the function to be built during that weekly cycle is healthy, but the Evo method is focused on systematic, weekly, measured delivery towards long-range higher-level objectives, within numeric, multiple, resource constraints. This means that the Evo method is more clearly focused on the wider stakeholder-set values, and on total resource cost management.

   The Evo method is not focused on simply writing code ('we are programmers, therefore we write code'). The Evo method is focused on delivering useful results to an organically whole system. We reuse, buy or exploit existing code just as happily as writing our own code. We build databases, train and motivate users, improve hardware, update telecommunications, create websites, improve the users' working environment, and/or improve motivation. So we become more like systems engineers ('any technology to deliver the results!'), than programmers ('what can we code for you today?').

   Table 1 shows the use of an Impact Estimation table (Gilb 2004) to plan and track critical performance and cost characteristics of a system (Illustration courtesy of Kai Gilb). The pair of numbers in the three left hand columns (30, 5 etc.) show initial benchmarks (30, 99, 2500) and Goal levels (5, 200, 100,000). The '%' figures are the real scale impacts (like 20) converted to a % of the way from benchmark to the Goal levels (like 20% of the distance from benchmark to Goal).

   	Step 12				Step 13
   	Buttons.Rubber				Buttons.Shape and Layout
   	Estimate		Actual		Estimate		Actual
   Goals User-Friendliness.Learn 30 min<->5 min by one year	-10	33%	-5	17%	-5	20%	5	-20%
   Reliability 99 days <-> 200 days by one year	-3	-3%	-1	-1%	20	20%	2	2%
   Budgets Project-Budget 25000 <-> 1000000 by one year	2000	2%	2500	3%	1000	1%	1000	1%

   Table 1. An Impact Estimation Table

7. Implement the project in Evo steps and report your progress after each Evo step.

   Report to the project sponsors after each Evo step (weekly, or shorter) with your best available estimates or measures, for each performance goal and each resource budget.

   • On a single page, summarize the progress to date towards achieving the goals and the costs incurred.

   • Based on the numeric feedback and stakeholder feedback, change whatever needs to be changed to reach the goals within the budgets.

   All agile methods agree that the development needs to be done in short, frequent, delivery cycles. However, the Evo method specifically insists that the closed loop control of each cycle is done by:

   • making numeric pre-cycle estimates,

   • carrying out end-cycle measurements,

   • analyzing deviation of measurements from estimates,

   • making appropriate changes to the next immediate planned cycles,

   • updating estimates as feedback is obtained and/or changes are made,

   • managing stakeholder expectations ('this is going to late, if we don't do X').

   The clear intention to react to the feedback from the metrics and to react to any changes in stakeholder requirements is a major feature of Evo. It helps ensure the project is kept 'on track' and it ensures relevance. It is only by the use of stakeholder metrics that Evo is allowed to have such control.

   Figure 2 shows results being cumulated numerically step by step until the Goal level is reached. In a UK Radar system (Author experience), the system was delivered by gradually building database info about planes and ships, tuning recognition logic, and tuning the radar hardware.

   

   Figure 2. Cumulation of results towards goal level

8. When all the goals are reached: 'Claim success and move on' (Gerstner 2002). Free remaining resources for more profitable ventures.

   A major advantage of using numeric goal and budget levels, compared to 'a stream of yellow stickies from users' (a reference to agile method practice), is that it is quite clear when your goals are reached within your budgets. In fact, 'success' is formally well defined in advance by the set of the required numeric goal and budget levels.

   Projects need to be evaluated on 'performance delivered' in relation to 'resources used'. This is a measure of project management 'efficiency'. When goals are reached, we need to avoid misusing resources to deliver more than is required. No additional effort should be expended to improve upon a goal, unless a new improved target level is set.

Project Policy

1. The project manager, and the project, will be judged exclusively on the relationship of progress towards achieving the goals versus the amounts of the budgets used.

   The project team will do anything legal and ethical to deliver the goal levels within the budgets.

   Projects need to be judged primarily on their ability to meet critical performance characteristics, in a timely and profitable way. This cannot be expected if the project team is paid 'by effort expended'.

2. The team will be paid and rewarded for benefits delivered in relation to cost.

   Teams need to be paid according to their project efficiency, that is by the results they deliver with regard to the costs they incur. Even if this means that super efficient teams get terribly rich! And teams that fail go 'bankrupt.'

   When only 13% of 1027 IT projects are 'successful' (Taylor 2001), we clearly need to find better mechanisms for rewarding success, and for not rewarding failure. I suggest that sharp numeric definition of success levels and consequent rewards for reaching them, is minimum appropriate behavior for any software project.

3. The team will find their own work process and their own design.

   Agile methods believe we need to reduce unnecessarily cumbersome corporate mandated processes. I agree. They also believe in empowering the project team to find the processes, designs and methods that really work for them locally. I heartily agree!

   However, I also strongly believe that numeric definition of goals and budgets, coupled with frequent estimation and measurement of progress, are much-needed additional mechanisms for enabling this empowerment. The price to pay for this, a few estimates and measures weekly, seems small compared to the benefits of superior control over project efficiency.

4. As experience dictates, the team will be free to suggest to the project 'sponsors' (one type of stakeholder) adjustments to 'more realistic levels' of the goals and budgets.

   No project team should be 'stuck' with trying to satisfy unrealistic or conflicting stakeholder dreams within constrained resources.

   Further, a project team can only be charged with delivering inside the 'state of the art' performance levels at inside the 'state of the art' costs. Exceeding 'state of the art' performance is likely to incur 'exponential' costs.

Summary

A number of agile methods have appeared, trying to simplify project management and systems implementation. They have all missed two central, fundamental points; namely quantification and feedback.

Evolutionary project management (Evo) uses quantified feedback about critical goals and budgets. Further, Evo also insists that early, frequent, small, high stakeholder value deliveries (Evo steps) are made to real users: this is only possible if supported by stakeholder metrics.

It is the use of stakeholder metrics that allows better focus, more measurement of progress, and more flexibility to change. It is time agile methods adopted quantified, critical stakeholder metrics.

References

Abrahamsson, Pekka, Outi Salo, Jussi Ronkainen and Juhani Warsta, Agile Software Development Methods. Review and Analysis , VTT Publications, Espoo, Finland, 2002, ISBN 951-38-6009-4, URL: www.inf.vtt.fi/pdf/ , 107 pp.

Gerstner, Louis V. Jr., Who Says Elephants Can't Dance? Inside IBM's Historic Turnaround , HarperCollins, 2002, ISBN 0007153538.

Gilb, Tom, Principles of Software Engineering Management , Addison-Wesley, 1988, ISBN 0201192462.

Gilb, Tom, Competitive Engineering: A Handbook for Systems & Software Engineering Management using Planguage , See www.Gilb.com for draft manuscript, 2004.

Jarzombek, S., Proceedings of the Joint Aerospace Weapons Systems Support , Sensors and Simulation Symposium, Government Printing Office Press, 1999. Source Larman & Basili 2003.

Johnson, Jim, Karen D. Boucher, Kyle Connors, and James Robinson, " Collaborating on Project Success ," Software Magazine, February 2001. www.softwaremag.com/L.cfm?Doc=archive/2001feb/CollaborativeMgt.html

Johnson, Jim, " Turning Chaos into Success ," Software Magazine, December 1999. www.softwaremag.com/L.cfm?Doc=archive/1999dec/Success.html

Larman, Craig, Agile and Iterative Development: A Manager's Guide , Addison Wesley, 2003.

Larman, Craig, and Victor Basili, " Iterative and Incremental Development: A Brief History, " IEEE Computer, June 2003, pp 2-11.

Taylor, Andrew, " IT projects sink or swim ," BCS Review, 2001. http://www.bcs.org.uk/review/2001/articles/itservices/projects.htm