How to live without estimates

"Estimations aren't right."
"Estimations are evil."
"Estimations are a necessary evil."
"Estimations are neither good nor bad."
"Estimations are hard."
"Estimations are rarely correct."
"No one enjoys doing estimations."

If estimations don't work then what to do instead? How to work? How to coordinate? How to plan work?

I will attempt to answer these questions.

We are so hard-wired to do estimations that it's hard to think about projects without them. We invented story-pointing, planning pokers, and butt-scratching but none of that seems to work. We don't like doing them and very often miss targets. They are not satisfying our work model or completing the puzzle that we are trying to solve.

When something, no matter how popular it is, doesn't look like a solution to the problem, what I do is altogether eliminate the idea and see what breaks. How can we live without estimates? How can we reorient the rest of the things to eliminate estimations? When something has a track record of not working, then there is no point in trying to adjust it.

That is how knowledge grows too. When something doesn't fit in the "model" you have to eliminate it and not compromise the entire model. From afar it might look like a project or a company is running smoothly but when you zoom in you will find these little black holes that suck the energy out of the company. Those black holes are the gaps in the model. I read about this idea of how knowledge grows in a recent essay by Paul G. It's a good but lengthy read.

Estimations, we know, is an Achilles heel but since we don't know what else to do we tolerate it as a necessary evil or come up with intellectually lazy alternates that are story pointing, planning poker, etc.

There is also a possibility that what is wrong is a side-effect of another wrong idea. An effect of a cause. By eliminating something wrong, we might discover other evils. This nested search of eliminating what is provably wrong takes us to first principles and reorients our thinking. It's a good way to grow anything. Personality, product, crops, or a company.

The same can be done with estimations. If we remove estimations, what would happen? Let's entertain a few cases.

We estimate so we can hold people accountable for their estimates and review their performance.

My argument is that if even the sharpest can't estimate correctly, then that means we are reviewing people on the wrong scale. Such a scale can't be trusted. Talking about scales, Wittgenstein’s Ruler implies:

When using a scale to measure something you could be measuring the credibility of the scale more than what is being measured.

Having that said, I am quite sure that the credibility of estimations as a scale to measure performance is pretty low. So, what do we do instead? We need other methods to measure performance. Thankfully, we aren't put out of options. Right now, I won't discuss what other methods are available. That would spiral out on a different tangent. If we just remove estimations from the arsenal to hold people accountable we can still carry on. Adding something in place of estimations is not a necessity. If there are some people who can only be held accountable based on their estimations, we need to get rid of them as well as the manager who thinks so. More on hiring and firing here.

Let's discuss another case of why we need estimations. We need to tell a customer when a certain bug will be fixed.

First of all, if the cause of the bug is unknown, we can't reliably estimate how long it will take to debug it. You theoretically can't give an estimate to a customer before knowing the root cause. If, unfortunately, you plan to duct tape then you might be able to estimate.

But what do we estimate? Let's say, there is a typo. do we really need to tell the customer, when a typo will be fixed? If it is that trivial of a fix but also harmless, we shouldn't even spend 30 seconds thinking about when will it make to production.

Let's imagine a serious case. Let's say we have a critical failure, and we need to tell people about it. First of all, develop processes to ensure that critical failure doesn't happen. Have monitoring and observability to debug it fast but when that happens, we should spend all our energies trying to fix it rather instead of estimating. The best we can do is to keep customers posted about the progress transparently about what is being done. Whether you are debugging or a cause has been identified or a fix is being done or released, etc. For critical failures, estimation is harder and even more of a waste of time.

Let's imagine a scenario that is of a bug of normal urgency but a fix will take time and it needs to be communicated because the customer has to do some planning around that. What do we do? We get the estimate from the person who is working on it. Add a generous buffer, add the time of the intermediate process (review, UAT, release cycle) before it can be released, and convey forward with a clear mention that it is an "estimate". The language used in these estimates is of weeks, months, and quarters and not days or hours. Give an estimate that you have a 99% chance of living up to and then delight your customer by shipping earlier. Such scenarios are/should be minimal.

I have worked with companies that do regular estimations and those that don't. From my experience and by observing other companies, I've concluded that the companies who do estimates and set deadlines regularly move slowly as compared to those who don't.

If I were to explain it in two words, I'd say: Parkinson's law.

The work expands to fill the time allotted for its completion.

Companies that do estimations and deadlines rigorously are more bureaucratic. Meeting deadlines is also used as a proxy of performance. In such a case when a manager (possibly CEO) asks a report that how long a certain task would take, the manager starts a chain reaction of reiterating the same question to his/her reports until it reaches the person who does the work. On each level of the chain, the estimate is bloated by adding a buffer to be safe. If a developer says, it will take 1 week, while it reaches the CEO, the estimate would be 1-2 months. Instead of delighting the customer, the whole game becomes about delighting the manager. This is a prime example of Principal-Agent conflict. CEO wants to improve the customer's life while employees want to live up to their performance metrics. Since the estimate by developer was already a conservative one, even if the work takes 2 hours to complete, due to Parkinsons law it will take 1 week from the developer's end. This is the reason such companies are slower.

Let's also consider what happens when the estimate was too tight for the work. In such a case, the developer would miss the deadline and possibly be frowned upon. The sprint would spill and the project manager's perceived perception gets hurt. Moving onward, both of them will add more and more buffers which slows down shipping further because of Parkinson's Law.

In the case when deadline was tight but still met, very often the developer, team, or manager will cut corners. It will end up with bugs in production or missing features. Fixing those bugs and features would still take the same amount of time and probably more in bureaucratic environments because people are busy playing (planning) poker instead of shipping.

If you eliminate planning pokers, deadlines, sprints, and estimates, you will ship faster. You will surface underperforming employees. You will minimize Principal-Agent conflict. You will improve culture.

Building a company's culture is all about minimizing the Principal-Agent conflict. Everything else is a cop-out.

Some people, in an attempt to exploit Parkinson's law to their advantage, put deadlines to limit the scope of the work. It works and for some people helpful. That is not estimation though. That is something like, "I will ship this MVP by Friday, no matter what". In that case, you rarely put effort into estimations except for a very intuitive guess. There are other methods to cut scope as well. Deadlines are not the only way. You can ship in iterations. Avoid big rewrites. Put big features behind feature flags and release the MVP as soon as possible. Push code into production regularly.

The conclusion is that to live without estimates you have to stop doing them. Stop setting deadlines, hire better engineers and managers, hire fewer managers, minimize Principal-Agent conflict to build a high-trust environment, and ship in iterations.

You need to have a good bar for hiring and firing people as well as clients/customers. If you happen to work in a low-trust environment, try to make it better. This is a good starting point. It will greatly improve the quality of your life and the people you work with.

If you can't improve it, try it. If you still can't, opt out of such an environment. Such an environment is likely making more money off of employees than customers or has no growth prospects.

If you don't want to improve it, whether as a manager who asks for estimates or as an employee who needs deadlines to function, then grow up. Try to meditate and work on personal growth instead. Dabble with various art forms, follies, hobbies, and experiments until you understand the fabric of life.

What other scenarios do you think make estimations necessary?