Why software projects take longer than you think: a statistical model (2019)
> A reasonable model for the “blowup factor” (actual time divided by estimated time) would be something like a log-normal distribution.<p>Interestingly, we did extensive time tracking on a multi-year in-house software project and collected data comparing the estimated completion time of tickets with their actual time.<p>The software department was under a lot of pressure to improve their forecasting, and were somewhat despairing that their estimates were off by a factor of about 1.6 on average, and sometimes a factor of 10 or more. This persisted in the face of all attempts to improve calibration. Managers were worrying that developers had no idea how long a task would take and estimation was futile.<p>When we plotted the data, in all cases, the actual time was <i>very</i> accurately fit by a lognormal whose scale parameter was precisely the predicted completion time. That is, whether the tickets were predicted to take 1 hour, 3 hours, 13 hours, or whatever, the histogram of their actual completion times followed the exact same shape but with a corresponding scale change on the x axis.<p>This told me that the developers actually have a really good understanding of the class of problem they're dealing with when they start a task. But sometimes tasks have multiplicative factors that make them take longer than you expect. Sometimes the bug turns out to be two bugs, and so on. Based on this analysis, I urged them not to consider it a prediction failure when a ticket takes 10 times longer than expected; that's just a property of the lognormal distribution, and that estimate likely did a good job of reflecting all available information at the time they made it.<p>Instead of changing the estimates, I suggested that we pick a safety factor for external facing commitments that reflects this distribution. Padding the estimate by 1.6 factor gives the mean, but if you want to make a commitment you can take to a customer, you can just extend the lognormal up to 95% confidence or 99% confidence or however trustworthy your promised commitment needs to be. Of course, a 99% confidence interval on a lognormal is a pretty big factor. But if that's better than running late, it is what it is.<p>Another interesting thing is that you'd expect, by the central limit theorem, that sufficiently large tasks would eventually become normally distributed rather than lognormal, because they're composed of a large number of subtasks. But it turns out that lognormals are a pretty pathological case; sums of n lognormals can continue to look nearly lognormal until n becomes really, really large.