If your software shop is working on such same-y tasks that they can be precisely estimated, your software shop is leaving immense amounts of money on the table in the form of something repetitive that ought to be refactored and automated away.

Beyond question, there are shops that operate this way. However, trying to translate the lesson on how to schedule out of this context into the context of a higher-end shop that can and will so automate is not possible.

Sure, automation is a smart move. But by same-y I mean:

* We need a website. It will have these 5 pages. How long will that take?

* We need an iphone application. Here are the dozen or so screens and a few pages of what it needs to do. How long will that take?

Automation is actually a diminishing multiplier on the estimate -- it doesn't make estimation itself as a problem go away. If anything it reduces variance, making estimates more useful.

Take the website example. It might have been that originally you did everything by hand but now you have a collection of templates, generation tools, snippets and so forth. This has brought your development time down substantially -- but you still get asked for estimates. What used to take weeks might now take days; fine.

But the end product still has some degree of size and therefore takes some amount of time to develop, even if that size is a hundred thousand lines of code and the time is however long it takes for your code generator to spit out Generico Inc's new website from the you-beaut inhouse system.

If you are making a 5-page website and you have effectively no code to write, the reason you can make an estimate cleanly is that it is no longer programming. At that point you are estimating how long the content will take to produce, get signoffs, graphics, etc, and those are quite estimable. The worst case you're looking at there is CSS incompatibilities. An iPhone app that is just layout out some UI elements and doing something very simple (or just pulling up existing webpages) will similarly mostly not be programming.

Of course if you remove most or all of the programming elements, the programming no longer screws your estimates up.

There's a whiff of No True Scotsman in my post here, so let me nail this down a bit more and say that in this case I see programming as requiring some sort of logic in it. Laying out a form for your iPhone app may occur in a programming tool and with another definition may even be part of the "programming", but for the purposes of this post I am not including that. And while having, say, a conditional panel that only appears when certain things is true is indeed programming, if your project is dominated by content and forms and just has a trace of logic, your exposure to the chaos of programming is limited and negligible. It's a continuum, of course, not a binary thing.

I think it was good of you to cite No True Scotsman.

I still think any estimate, even hilariously broad ones, are useful. Watts Humphreys put it this way: when is an estimate going to be most accurate? When you've just finished. When is it most useful? When you're just starting. In between is a tradeoff between usefulness and accuracy. You accept that estimates are inaccurate -- that's why they're called 'estimates'. But even if you narrow estimation variance from +/- 400% to +/- 200% over time, that is still a valuable improvement.

When I'm working with clients I explain the cone of uncertainty and so far they've all been understanding of the fact that software work contains profound uncertainties that other lines of work won't.

In my anecdotal experience, at large corporations with well-defined processes, the 5% variance is the result of huge time estimates. Then you make sure you don't implement any faster than the estimate. (Even if it turns out you could have done it half the time.)

I was thinking something similar. In my anecdotal experience [1], large corporations can control variance by normalizing everyone's velocity. For example, in order for me to finish a feature, I had to write a document describing how I would solve the problem and convince my manager and tech lead that it would work. After spending about two months getting my plan approved [2], they let me implement it, which took me about four hours. At that ratio, the actual development time is practically negligible, so it's easy to have low variance. The side effect is that you also have very low productivity.

Imagine the opposite scenario, where they let me implement it first and then approve it if it works. Let's say it takes me a couple days to finish it (this time has to include some of the time that it took me to figure out how to solve it originally). Now the feature is getting done in a matter of days instead of months. The flip-side is that you can have higher variance, because if there's a problem with my initial implementation, that adds more time to fix the problem. In this scenario, the time it takes to finish features is more dependent on the engineer's ability than the time it takes to pass through the bureaucracy, and there's going to be more variance in the abilities of individual engineer.

[1] I worked at Microsoft as an intern in the summer of 2009.

[2] There were many days that I literally couldn't do anything except wait for clearance to go forward.

You're thinking of Parkinson's Law -- another well known finding in all industries.

A lot of comes down to how estimates are being used. A lot of places use estimates as the plan rather than ... an estimate. And that's where stuff goes awry, because as you say the incentives are to stick to the plan even if it's stupid.