Irrational Optimism and Project Planning

Mike Perkins

I read an article in the Wall Street Journal a couple of months ago that resonated strongly with me. Although the article deals with the problem of project planning for infrastructure projects, it’s really about irrational optimism. Why is it that infrastructure projects always cost more and take longer than expected? (As anyone reading this post knows, this problem isn’t limited to infrastructure projects!)

Here is one of the insights:

[T]he pattern of budget excess echoes findings in other areas where people allow their best hopes to dominate the planning process. Irrational optimism afflicts even individuals who have experience with a given situation, and should know better. Researchers have linked it to business analysts’ earnings forecasts and to students predicting when they will finish assignments…

Magne Jørgensen, who has studied persistent underestimates of completion times in software development, says businesses would do well to maintain data on prior forecasts. “Looking forward makes you more optimistic,” says Prof. Jørgensen… “Looking backward makes you more realistic.” (italics mine)

Cardinal Peak has always strived to make realistic estimates for its customers, and over time we have become better at doing so. Mike Deeds and I, who often end up estimating projects together, have found a technique that seems to work pretty well. We start with a bottom up estimate: We create a list of the tasks that need to be performed; we estimate the time required to complete each task; and we multiply the aggregate time by 1.2.

The “1.2 factor” helps to account for two things. First, the need for interaction among developers to complete a task—after all, system components, whether hardware or software, never exist in isolation, and interaction takes time. Second, the fact that developers often encounter unforeseen challenges—things that are hard to know when planning and are only learned when you sit down and start actual design and coding.

Next, we step back and ask ourselves which project that we’ve done in the past is most similar to this one. Once we’ve identified it, we ask ourselves how many engineers were working on that project and how long it took them. We make any obvious adjustments to account for the differences between the projects, and if after that, the bottom-up and top-down estimates are not within 10% of each other, we dig in and ask why. Have we missed some obvious tasks (e.g. debugging time)? Have we forgotten how long it takes to do QA and fix bugs in the real world? Have we underestimated how long it will take to develop manufacturing tools and support a handoff?

Unfortunately, convincing customers that our estimates are truly realistic is sometimes challenging, especially when the estimates are higher than a customer had hoped. We understand this and do our best to help them understand that our estimates seem “high” because they are realistic and were generated by experienced engineers. The WSJ article notes that:

[R]esearchers say another factor exacerbates the effects of irrational optimism: Projects planned most optimistically will look more attractive to funders than those that are planned realistically. So even if not every planner is optimistic, those who are have the best shot of moving forward… “Optimism is tremendously inspiring,” says Dale Griffin, professor of marketing… “Accuracy, perhaps not so much.”

Over time we’ve also learned that clients remember the lowest number that was ever placed in front of them, regardless of how early in the planning process it was generated, and how many feature requests and changes they have made since then! The best way we’ve found to deal with this is to resist the temptation to throw out a number before the requirements are understood, and to insist on a solid requirements document and payment for the time needed to generate a plan. Generating a good plan is a lot of work; real engineering occurs during the plan generation process.

Sometimes, when the pressure to get overly optimistic feels particularly intense, we have to remind ourselves that those who make overly optimistic estimates frequently end up suffering from what the WSJ calls “the consequence of rosy projections”. We’ve been there too, and it isn’t pretty.

Mike Perkins, Ph.D., is a managing partner of Cardinal Peak and an expert in algorithm development for video and signal processing applications.

Cardinal Peak
Learn more about our Audio & Video capabilities.

Dive deeper into our IoT portfolio

Take a look at the clients we have helped.

We’re always looking for top talent, check out our current openings. 

Contact Us

Please fill out the contact form below and our engineering services team will be in touch soon.

We rely on Cardinal Peak for their ability to bolster our patent licensing efforts with in-depth technical guidance. They have deep expertise and they’re easy to work with.
Diego deGarrido Sr. Manager, LSI
Cardinal Peak has a strong technology portfolio that has complemented our own expertise well. They are communicative, drive toward results quickly, and understand the appropriate level of documentation it takes to effectively convey their work. In…
Jason Damori Director of Engineering, Biamp Systems
We asked Cardinal Peak to take ownership for an important subsystem, and they completed a very high quality deliverable on time.
Matt Cowan Chief Scientific Officer, RealD
Cardinal Peak’s personnel worked side-by-side with our own engineers and engineers from other companies on several of our key projects. The Cardinal Peak staff has consistently provided a level of professionalism and technical expertise that we…
Sherisse Hawkins VP Software Development, Time Warner Cable
Cardinal Peak was a natural choice for us. They were able to develop a high-quality product, based in part on open source, and in part on intellectual property they had already developed, all for a very effective price.
Bruce Webber VP Engineering, VBrick
We completely trust Cardinal Peak to advise us on technology strategy, as well as to implement it. They are a dependable partner that ultimately makes us more competitive in the marketplace.
Brian Brown President and CEO, Decatur Electronics
The Cardinal Peak team started quickly and delivered high-quality results, and they worked really well with our own engineering team.
Charles Corbalis VP Engineering, RGB Networks
We found Cardinal Peak’s team to be very knowledgeable about embedded video delivery systems. Their ability to deliver working solutions on time—combined with excellent project management skills—helped bring success not only to the product…
Ralph Schmitt VP, Product Marketing and Engineering, Kustom Signals
Cardinal Peak has provided deep technical insights, and they’ve allowed us to complete some really hard projects quickly. We are big fans of their team.
Scott Garlington VP Engineering, xG Technology
We’ve used Cardinal Peak on several projects. They have a very capable engineering team. They’re a great resource.
Greg Read Senior Program Manager, Symmetricom
Cardinal Peak has proven to be a trusted and flexible partner who has helped Harmonic to deliver reliably on our commitments to our own customers. The team at Cardinal Peak was responsive to our needs and delivered high quality results.
Alex Derecho VP Professional Services, Harmonic
Yonder Music was an excellent collaboration with Cardinal Peak. Combining our experience with the music industry and target music market, with Cardinal Peak’s technical expertise, the product has made the mobile experience of Yonder as powerful as…
Adam Kidron founder and CEO, Yonder Music
The Cardinal Peak team played an invaluable role in helping us get our first Internet of Things product to market quickly. They were up to speed in no time and provided all of the technical expertise we lacked. They interfaced seamlessly with our i…
Kevin Leadford Vice President of Innovation, Acuity Brands Lighting
We asked Cardinal Peak to help us address a number of open items related to programming our systems in production. Their engineers have a wealth of experience in IoT and embedded fields, and they helped us quickly and diligently. I’d definitely…
Ryan Margoles Founder and CTO, notion