A common service level agreement (SLA) measurement among Information Technology Infrastructure Library (ITIL) practitioners is project completion time, in one form or another. Many use a percent difference measurement between actual and forecasted, others simply subtract the actual completion time from the forecasted completion time. Whatever the final calculation, using the elapsed time between when a project starts and when it ends cannot be escaped. It is a critical measurement to take accurately because it is used by most project management software packages for future forecasting as well as internal operational analysis.

A measurement system analysis (MSA) is used by professional process engineers to ensure that a measurement system is measuring accurately – with reasonable variation. With an acceptable measurement system, managers can look at 20 project completion times on a spreadsheet and have a high degree of confidence that those numbers accurately reflect – with reasonable variation – what is actually happening within their organization. Without a solid MSA, that same manager could not tell with reasonable certainty if differences in completion times between Project A and Project B were due to an inaccurate measurement system, or other, more meaningful factors that should be addressed. Similarly, variation in the measurement system can trigger management action where none was needed.

Challenges for Completion Time MSA

Two factors challenge conducting an MSA for project completion time. First, the measurement is destructive, which is to say that no one can rewind time and experience project events a second time exactly as they occurred the first time through. The opposite of this, for example, would be the ability to measure the diameter of a manufactured hub cap. The same hub cap could be measured by one person, then a second and finally a third to see if the three are measuring it differently. Obviously this cannot be done with project completion time – once it is done it is done.

The second is that measurements from several different projects are not homogeneous, which is to say that each project is unique to a large degree. Continuing the hub cap example, a set of homogeneous data would result if one took one hub cap, a second hub cap, a third hub cap, and so on through a tenth hub cap, from the same manufacturing line and recorded diameter readings from all 10. The expectation would be that all 10 would be reasonably similar since the same (or nearly the same) conditions produced each. Not so with systems engineering and software development projects producing a completion time measurement. Though each project may have general similarities they are, due to their design nature, significantly unique.

To overcome these challenges, those who measure completion time must first acknowledge that the actual length of time for different projects will, of course, be legitimately different. However, the measurement’s calculation consistently depends upon two independent factors – start time and stop time. What does – or should – not change between projects, and what should be applied consistently to ensure clean completion time data, is the criteria which determines and triggers the entry of a project’s start time and stop time. A simple example would be timing a sprinter: When does the timer start the stopwatch and when do they stop it? Everyone knows the timing starts at the bang of the starter’s pistol and steps when the runner breaks the plane of the finish line. So then, management must clearly develop, articulate and communicate the start-stop criterion. And, management must remember that the stopwatch should be started and stopped from the customer’s point of view.

Now, how can management test to ensure that the start and stop criteria are well understood and applied correctly and consistently throughout the organization? As indicated, no one can rewind time and replay a particular situation to determine if everyone would record the project start and stop time identically, and is equally impossible to replay the situation for the same person a week later to see if, when faced with the same situation, they would make the same determination on start and stop times.

Using a Number of Real World Scenarios

What can be done, however, is to write a short, realistic scenario based upon real events as they typically unfold in the organization. To make the test a bit more robust, management can quickly devise 10 realistic scenarios covering a wide range of observed real world events. If the scenarios are written so that times and dates are associated with key events, management could then ask practitioners to read the scenarios, apply the criteria and record their judgment as to when the project started and stopped. The practitioners could then be asked to re-read the scenarios two weeks later, having reshuffled the order in which the scenarios are presented, to see if the criterion is repeatable. If there are significant variation in start or stop times – either between different practitioners in the organization or between the same practitioners over time – then the measurement system needs improvement (the criteria is not clear), and management can have little confidence in project completion time numbers presented on official reports.

Here is an example of just one such scenario and resultant data collection in Tables 1 and 2:

Looking at Trial 1, although Practitioners 1 and 2 agreed on the project’s start time, their choices did not match what the organization defined as the project start time from the customer’s viewpoint. Also, there was disagreement between the practitioners as to the project’s stop time, although Practitioner 1 correctly matched the organization’s criteria.

Looking at Trial 2, Practitioner 1 has different responses for both start and stop times. Practitioner 2 responded differently to the start time and was consistent with the stop time.

This variation between trials and practitioners clearly indicates that the organization does not readily understand when a project should officially start or end. Obviously, a situation needing correction given the importance of such a measurement. Correction typically takes the form of better articulation of the criteria coupled with quick, informal training. This training is a great vehicle for feedback to management, as many “what if” scenarios typically surface. Keep in mind also that this is just one scenario for illustration. In practice, multiple scenarios should be employed to gain reasonable coverage – typically 10 scenarios is the number recommend. Also, three practitioners are generally used.

Impact of Project Completion Time

What is the impact on project completion time? Table 3 shows an example, using straight calendar days for simplicity:

Which value for project completion time is an IT or business manager to trust? Keep in mind that the reality of the situation is identical for all of these values. The large amount of variation is due solely to the measurement system. Most would agree that management action also would vary as a function of which number was presented, reinforcing the need to get it right. 

Management by fact is critical to organizations utilizing ITIL, capability maturity model integration, business process management, Lean Six Sigma and similar frameworks. A software or IT organization will rarely have all of the information its wants or needs, so making sure that the facts it does have are accurate is paramount. No one would dream of flying over the Rocky Mountains in an airplane with un-calibrated altimeters, yet some organizations settle for this situation when managing their business. Clean data is essential for making sound decisions in both situations. It is clear that conducting a measurement system analysis to understand the reliability of organizational measurements is both easily accomplished and well worth the small investment.