How to plan for software project estimates that are almost certainly wrong

At the start of a software delivery, there is a lot of uncertainty in regards to what will be delivered. Since we can't know it all, any predictions made will be incorrect. But those predictions are correlated and consistent, in a way that can give us useful insight. In this piece, we look at how you can plan on the basis of these incorrect predictions and why you should embrace them.

We estimate because the future is unknown

Why do we estimate software projects? It's a reasonable question given that software estimates have a track record of being wrong. One section of the software community would argue that you shouldn't bother because they will never be accurate. But not us! We are the estimation optimists! We believe that good estimates allow an organisation to make commitments they can keep. Resources (meaning people), budget (meaning how much those people cost) and scope (meaning what those people produce) all need to be managed ... somehow. If you work for an organisation that doesn't try to make and then keep commitments around resources, budget and scope... then move along - this post isn't for you. For the rest of you, read on.

Art versus science in estimates

One of the seminal works on estimation[1] introduces us to two competing tracts in the world of estimation. On the one hand the science of estimation attempts to improve estimates from within ±10% of the eventual actuals to ±5%. This technique uses ever more sophisticated mathematical techniques in search of gains in a world of near perfect input data and well understood expected outputs. He contrasts this with the second tract where typical software organisations are trying to improve on estimates that are often 100% wrong! In these scenarios, it's not the science of estimation that matters, but rather pragmatic approaches to improving predictions. It is this art of software estimation that this article is concerned with.

First order approximations are good enough

Zeroth order approximations are very rough guesses, where the only number you specify, is itself, only an approximation. For example a few thousand trees doesn't even tell you how many thousands of trees exactly, let alone a specific number of trees. First-order approximations give a more accurate picture because at least one number is given. For example approximately 4 thousand trees, or a 10 week project. Second order approximations are more accurate still, with two or more significant figures given, for example 3.8 kilometres or 12.2 months. The higher the order, the more precision is required to measure it, and the smaller the effect of each subsequent adjustment in precision, to the overall measurement.

When we are trying to predict the future using the art of estimation... we are typically in a situation where a first order approximation will be good enough. At the start of the project no-one is asking for the exact minute or hour a deliverable is going to be ready. They probably aren't critically sensitive (yet) the actual day. What they do need to know is when it will be ready to the nearest week because being off by a week will have material consequences in terms of budgets and resources.

Prepare to embrace your incorrectness

How much does a new building cost? It depends on what sort of building it is. If it's one more identical house to hundreds of others that a builder has built before then the estimate will be pretty good. If its a one-of-a-kind Scottish parliament, the estimate might well turn out to be pretty bad [2]. The same is true for estimating software. If it's something that has never been built before then there will be a lot of uncertainty around what it is actually going to take to complete the work. That uncertainty is at its absolute high point before any of the work has even started... but that's usually the point where the organisation has to decide if it is willing to commit to the project!

The cone of uncertainty

We make estimates because the future is unknown. We look at what we expect to happen, and then use those expectations to build our estimates. We know that sometimes estimates won’t match reality. One thing we do expect though, is that are level of uncertainty will reduce, as the project proceeds. Welcome to The Cone of Uncertainty!

The cone of uncertainty describes how the variance in estimates will (or could if you actually re-estimated) decrease as you get further along in the project and the future becomes more knowable. A big part of this is that software projects are a myriad of thousands upon thousands of micro-decisions. Decisions about scope, features, implementation, techniques, capabilities, interdependencies etc. etc. At the left of the cone very few of these decisions have been made. At the right, most of them have been.

Whether the variance is actually ±400% or not is less relevant than the fact that in the early phases of a project the estimates will be less accurate than later on. You hear that? Will be less accurate. The cone represents the best case! It's certainly possible to be less accurate but it's not possible for your estimates to be more accurate than the stage of the cone they are made at. An estimate can be more lucky, but it can't be more accurate.

Approximately correct is our target

Remember earlier when we said that first order approximations were good enough? That's what lets us sleep at night as project managers. Your estimates are going to be incorrect. And that's ok. The question is whether we can make them approximately correct, and more specifically, increasingly approximately correct as the project proceeds? Why is that our target? Because the main goal of a project manager is to be able to flag if the plan needs to be adjusted and offer the inputs to that adjustment process. Even if the estimates had been completely accurate, tehre were too many uncertainties around scope at the outset to allow the plan to stick. At some point the plan is going to change, and when it does, a project manager needs to be approximately correct in how those changes will impact the overall delivery.

Our estimates are correlated and consistent

Studies from as far back as 1991[3] show that developer estimates are typically 20-30% lower than their actual effort. As seasoned project managers joke, a developer estimate is the first non-zero probability completion date for the work! The good news is that across a series of estimates from the same developer, any underlying optimism is likely to be consistent rather than chaotic. Similarly, estimates for a group of requirements made by a cohort of developers are likely to be intrinsically consistent rather than chaotic.

Count then compute

Imagine you have to estimate the volume of marbles in a jar filled with a random selection of marbles. Each marble is either 0.5, 1 or 1.5 cubic centimetres in volume. You can either a) eyeball it and have a guess; b) calculate the volume of the jar and multiply it by the percentage you think the marbles occupy of that volume; or c) tip out the marbles, count how many of each size there are and multiply by their volumes. Clearly the 3rd option would give the best estimate. This holds true whenever you are estimating - it's really hard to eyeball more accurately than simply counting how much of a thing there is.

Given that estimates tend to be consistently wrong, this leads to the conclusion that the largest variance in overall project planning is likely to be introduced if we simply forget to estimate a part of the work. Choosing what to count depends on the stage of the project but as a minimum counting requirements is always going to outperform broad brush project-level estimation. Choosing how to count is a matter of refinement. Remember, not all marbles are the same size, so it is going to be better if you count relative sizes of things because the amount of effort for each thing will correlate to some underlying factor. This is the basis of story point estimation, where developers are asked to consider relative complexity of work because it uses the assumption that complexity correlates with effort, but it works for other techniques too.

Delta scope

So what does correlation and consistency get you? It gets you the knowledge that your estimates are almost certainly wrong, but they are wrong in a way that is consistent to other projects. It also gives you the knowledge that if you out your effort into breaking down the work into reasonable chunks, and work hard not to miss something out... you are more likely to get a better estimate. Furthermore, it tells us that as a project gets underway if we can work out how in what way our estimates are wrong, we can use our knowledge of consistency to our advantage to improve our estimates by predicting deltas for the scope that remains to be completed. Finally, it tells us that the largest variances in delta scope is likely to come from work that was never in the plan in the first place and if we've worked hard up front this is likely to be because it wasn't missed... it's simply something new we are being asked to do.

Back to the future – predictions for your software delivery

One of the things that is obvious but still powerful is that by far the biggest signal in project estimation is knowing which tasks are complete or incomplete. We know that hundreds of task management and todo lists are based on the premise that knowing that something is not yet done is the only bit of data that is relevant. For software projects that's also largely true. But not quite. Unfortunately in the world of software are marbles can vary quite a lot in size, and sometimes an extremely large marble is really just an indefinite number of smaller marbles waiting to be worked out. So while getting things done is indeed the major signal in a software project, we can do better, a lot better, than simply counting how many things are left to do, or even counting the original estimates on the outstanding items.

One approach is to constantly re-estimate and refine our estimates as we gain more information, moving along the cone of uncertainty and gradually getting things done. A team that is willing to re-estimate on a weekly basis is likely to end up with some very accurate estimates as the project proceeds (although they might not actually be spending much time doing the work!)

An alternative is to use the delta scope we've observed so far and use that to predict the delta scope of the remaining work. That's the basis of a dynamic contingency and it can be a powerful tool for getting better estimates with low effort as a project proceeds.

Benefits of historical data

But what do we do at the start of the project when we are at the left of the cone of uncertainty and we have little or no delta scope data in this project? Again we lean on our old friends correlation and consistency. It's not true that any project will have a historical doppelganger that we simply need to unearth to act as a roadmap for our new piece of work. But it is true that previous projects will cluster together in ways that have higher or lower consistency and correlation with our new project. The trick however is 1. having access to this type of historical data and 2. finding the best fit cluster to the current project. For decades larger software companies have known of the benefits of gathering and tracking historical estimation data to use for future projects. Its just always one of those things that never really seems to be a priority in 95% of companies. It's back to the science of estimation again - who really has time for that?

Outlier requirements

None of this is going to save you from an outlier requirement. That is, a requirement that falls well outside the range of predicted outcomes. We're not even talking about something that is several orders of magnitude outside the expected range... in a world relying on the 3 Cs of counting, consistency and correlation, even a 50% estimate error is going to break your prediction badly.

That's why one of the most important jobs a project manager can do is to detect outliers early and re-plan taking them into account as soon as possible. Its a much better outcome from an estimation perspective to detect the absence of a chunk of work, highlight it, and re-plan on that basis. Protect the consistency and correlation signals you are getting from your estimates at all costs! It's literally the only thing separating you from a guesser!

Takeaway

Just as the organisation is trying to make a promise it can keep around resources, budget and scope, our job as project managers is to make and keep a promise around when we predict everything will be done. There’s a lot of uncertainty at the start of a software delivery and any estimates we make are going to be wrong. Instead of trying to avoid it, embrace it! Accept that the promise you make is actually a promise to keep reading the data and updating your current prediction. Planning and estimating software requires us to anticipate things that are out of our immediate control but we can learn to manage consistency and correlation in a way that makes us comfortable as we move long that cone of uncertainty.

  1. Software Estimation: Demystifying the Black Art by Steve McConnell.
  2. See the Wikipedia page for the Scottish Parliament Building to see just how bad...
  3. Why Is Software Late? An Empirical Study of Reasons for Delay in Software Development, IEEE Transactions on Software Engineering SE-17 no. 6 by Michiel Van Genuchten.