I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the Earth.
– John F Kennedy, May 25, 1961
The complexity for minimum component costs has increased at a rate of roughly a factor of two per year… by 1975, the number of components per integrated circuit for minimum cost will be 65,000.
- Gordon Moore, “Cramming More Components onto Integrated Circuits – Electronics Magazine, April 19, 1965
Countless industry studies over the past several decades have revealed a harsh truth about engineering: the vast majority of engineering projects miss their schedules. In fact, we are so accustomed to missing schedules that the slip has become an integral assumption within our industry and within the engineering community. Nobody blinks an eye when they hear that an engineering project is late. In fact, the question is no longer whether or not we were late, but by how much. Our profession has utterly and completely normalized the notion of missing schedules, and the entire technology sector has shrugged their cumulative shoulders and gone along with the game.
Chronic problems provide fertile grounds for memes, cliches, excuses, and truisms, and entire industries have built themselves on the assumption that engineers will forever be behind schedule. With the possible exception of weight loss, few issues have had more futile advice thrown at them. “You can’t schedule invention.” “Adding resources to a late project just makes it later.” “Nine women can’t make a baby in one month.” “Creeping enhancements cause schedule slips.” – We could populate pages and pages with tips, tricks, and quips, and yet the truth remains. It is apparently impossible to engineer to a schedule.
Or is it?
Interestingly, two of the most massive, famous, and impactful engineering projects in history nailed their schedules against seemingly impossible odds. JFK set a deadline for landing man on the moon, and engineers beat the schedule by almost six months. Gordon Moore set a schedule for improving transistor density in integrated circuits, and engineering tracked it for ten years, then tracked his next challenge for four more decades.
What do these two examples have (that 90% of the engineering schedules in the world are apparently missing) that allowed them to be completed on time? Did Kennedy and Moore build the all-time best PERT and Gantt charts? Were these projects waterfall? Agile? Did the world’s engineers come up with the most accurate and detailed task decompositions and breakdowns, pushing them up through levels of management hierarchy where they were assimilated and matched with marketing requirements, then disseminated for review and tracked with weekly stand-up meetings? Perhaps they benefited from open-office-concept work environments fused with telecommuting for geographically dispersed team members using state-of-the-art collaboration best-practices? Were there variable compensation schemes and lucrative management bonuses depending on MBOs and KPIs versus plan?
Ummm. Nope.
One thing that each of these projects had in copious quantities was vision. Vision is elusive. Vision is transcendent. Vision moves problem-solving into an elevated context. The moon landing was a task – a scheduled event. But the benefits and impact of that project extended far, far beyond the scope of putting a man on the moon. Landing on the moon accomplished almost nothing of value, but the successful execution of the moon landing project, and the technology and infrastructure developed enroute to that goal changed the world. Similarly, cramming more transistors onto a chunk of silicon is a relatively mundane, almost academic exercise. But the penumbra of that project – the utter enormity of the technology and life-changing advancements that have been sucked along in the wake of that task – is almost incomprehensible.
Both Kennedy and Moore were acutely aware of the potential impact of their vision. In “Cramming More Components…” Moore foresaw home computers, automotive telematics, electronic watches, cell phones, and the infrastructure of the internet – all in 1965:
Integrated circuits will lead to such wonders as home computers—or at least terminals connected to a central computer—automatic controls for automobiles, and personal portable communications equipment. The electronic wristwatch needs only a display to be feasible today.
But the biggest potential lies in the production of large systems. In telephone communications, integrated circuits in digital filters will separate channels on multiplex equipment. Integrated circuits will also switch telephone circuits and perform data processing.
Computers will be more powerful, and will be organized in completely different ways. For example, memories built of integrated electronics may be distributed throughout the machine instead of being concentrated in a central unit. In addition, the improved reliability made possible by integrated circuits will allow the construction of larger processing units. Machines similar to those in existence today will be built at lower costs and with faster turnaround.
But Moore didn’t ask engineering to create all of those “wonders.” He simply told us to cram more components onto integrated circuits. The rest, he must have reasoned, would take care of itself.
It would be fair to argue that neither Kennedy nor Moore created an actual engineering “schedule.” They proposed a goal and a timeline or, in the case of Moore, more of a prediction. Moore’s prediction was not based on analyzing the engineering work to be done; it was a simple extrapolation of historical trends combined with a key bit of insight – that those trends would be likely to continue for a long time due to an apparent lack of obstacles. Kennedy’s moon shot was nothing of the sort. His was clearly a goal with a target date – again, with zero analysis of the engineering work to be done.
Perhaps the problem with engineering schedules is that we try to engineer them.
I managed large software development teams for a couple of decades, and one thing I learned clearly was that engineers have a certain logarithmic sense of time when it comes to estimating individual tasks. I noticed that when I asked an engineer “How long will XYZ take?” I would most often get one of two answers: “a couple of hours” or “three or four weeks.” I began to see that the “couple of hours” case meant that the engineer already had a few lines of code in mind, there was no ambiguity to the task, and I was getting an estimate for the normal time it took to do an empty engineering change – find the correct file, drop in the code, build, test, document, check-in.
The more interesting case was the standard “three or four weeks” estimate. That, I came to learn, meant there was ambiguity. The engineer did not have an immediate solution in mind, and experimentation would be required. When I followed up on these tasks, I further noticed a common sequence of events. First, after only a day or two, I’d get the, “This is going much faster than I thought” update. Often, there would be a demonstration that showed the new feature kinda-sorta-working, with a disclaimer that there was still plenty of work to be completed behind the scenes. If I checked in again in a week or so, things would still be “coming along nicely,” and I’d be told that we were “probably 80-90% done.” Ah, we were still “ahead of schedule.”
At the end of the “three or four weeks,” though, things would have taken a sinister turn. I was generally now told that we were “basically done, except for a few last issues to clean up.” I knew what this meant. We had just entered a stable state for schedule slippage, where we would remain at the “95% complete” mark for a period of time that could be anything from one day to six months. This was not engineering malpractice. This was the harsh reality that most engineers are simply incapable of creating reasonable estimates for engineering tasks they have not yet attempted. Interestingly, I found that there was no feedback loop, either. At the end of the project, when a “three or four week” task had taken three or four months instead, I began closing the loop – asking how long particular tasks had taken. Consistently, I’d get back the original estimate – “seems like it took three or four weeks.”
Perhaps moon landings and Moore’s Law did not have a magic ingredient that my schedules were missing. Maybe it was the other way around. Perhaps there is a convergence, where the true visionary can intuit the size of a huge task more accurately than we can deduce it through exhaustive analysis and decomposition, and perhaps the engineering spirit of problem solving and inspiration can meet that halfway. If a deadline is simply another key requirement, missing the schedule is a first-order failure to solve the problem. If a team of talented engineers is challenged and inspired to solve a problem, something transcendent often happens.
If JFK had not made his speech, or if Moore had not written his article, would we have had moon landings and a silicon-driven technology revolution anyway? Or was that original vision a key catalyst that partnered with the engineering spirit to manifest something from nothing? Of course, we will never know the answer.
Late schedules are 99% the way we manage people and projects, with no skin in the game. Simple concepts, difficult to manage in some organizations. KISS, Mythical Man Month, Research vs Product Development, Project Definition vs Product Evolution, Staffing Skill Sets vs One Size Fits All, Accountability, and defined project/organizational feed back.
Nearly every project I controlled from 1973 to 2009 delivered early or on-time. Nearly all were fixed price contracts, with short schedules, and high degrees of uncertainty. Many were failed projects where the in-house team or contractor failed to deliver, often were the team had no plausible path to completion visible for their management (both technical and schedule). Those that were not fixed price contracts, were in-house projects where I was project manager on the payroll W-2 or 1099.
Meeting schedules was not an accident, or stroke of luck, or without incident. It was with a clearly defined framework that avoided a fair number of pit falls, plus heavily leveraged experiences of myself, my team, and other external teams.
The first fixed price contract I took in 1973 while a junior in college. I accepted a 3 month bid as a sub-contractor from someone I trusted, but had set me up to be the fall guy for an already failed project. I lacked the experience to know better. Completed the project out of stubbornness, a year late, earning less than $0.50/hr, with a lot of introspection at the end about what was wrong from the beginning. The client hired me W-2 for the next year at an exceptional salary, to finish the remaining two phases of the project that other two other contractors had defaulted on because they got carried away with the ground breaking research, and lost sight of actually delivering the product.
Rule 1: Fixed price contracts where if you fail to deliver on time there is a penalty (or lost incentives). Fixed price contracts come with invariant fixed deliverables, and payment for those deliverables. Design changes and enhancements are clearly the scope of the follow on project or milestone, NOT the current deliverable. This is possible in-house with W-2 staff, when both staff and management agree to stick to defined product development roadmap, development plans, and product evolution, as clear incremental deliverables. Keep development projects short, well defined, and invariant. Focus on this deliverable. Push change requests into phase two, phase three, etc deliverable requirements. Schedule internal completion deadlines well ahead of external deliveries, including mandatory slack/vacation time just prior to rollout, as there will be critical staffing demands to manage roll-out.
Rule 2: Strict KISS (keep it simple stupid) product management. 99.9% of your users/customers will only use the 90% feature set of the core product. That “other” 10% of “it’s nice to have features” will bury your development staff and force extensive rework to change the original implimentation framework with endless rewrites to “correct”. You can NEVER remove the candy features once a small group of vocal customers become attached to them … that WILL NEVER pay for the maintenance development costs they demand. The 90% of a project is completed in the first 10% of the time is real … it does take 90% of the time to complete the last 10% of less critical features, that will become a boat anchor for the products life.
Rule 3: Never give a product development team a research project, not even with clear goals and schedules. The project will always stay a research project chasing great ideas that are never completed. DO NOT let Marketing drive this with inter-departmental mandates or bartering. Do make Marketing an active voice in setting deliverables and schedules, because in the end they create your paycheck. Everyone can work behind a well defined product roadmap, no body is happy in ever changing quicksand that will devour your credibility and soul.
Rule 4: Test is everything, early and often … and 99% of that belongs to the developer writing the code. If developers throw shit over the wall to QA, the project has just failed. Only the developer clearly knows where all the decision/design points and edge cases are for solid unit test … it’s a failure to expect some QA test writer to see all of these critical points … and the bugs will slide quietly into releases. If your developers can not do solid unit test before handoff … they need to be working for your competitors. You will NEVER be able to dynamically assess your progress, when test is saved for the end … you will fail, you will schedule over-runs, and you likely will hate yourself for it as a manager or senior developer with a long history of failures.
Rule 5: One-size-fits-all is a Myth. People are individuals with unique skills, temperaments, and experiences … where given the same situation, will come up with different views of the problem based on their unique history. Productivity varies by an order of magnitude depending on the person, problem, and environment. I staff project teams with a visonary, a prototyper, closers, and a fire fighter. The first two generally have the skills and temperament to define concepts, produce a framework/architecture and fluidly evolve the early project to a demonstration prototype stage in the dark of night … this is rarely a finished product. It’s equally rare that they have the patience to morph the prototype into a long term supportable product, or even complete the design documentation for a product team to finish. That’s where the project needs to be handed off to a closer team, the people that actually define the deliverables, and see the project to product release. Somewhere near the end of the development cycle and QA cycle as the product rolls out to alpha, beta, and is released into the field it’s really useful to have a personality that can get to the bottom of critical failures, repair, and release them without creating a lot of drama or finger pointing. Fires happen … they need to be dealt with quickly, and nearly invisibly, including cleanup.
Rule 6: Accurate and complete project reviews … objective post-mortums. People can not learn from mistakes, when they are all swept under the rug trying to avoid accountability. Nor can solid reliable performers be rewarded for their contributions when the stage has been set with drama and credit takers. Mistakes are how we learn the hard lessons … own them. I do not hire developers that can not cite at least several critical learning points in their career … either they had no responsibility, or they were riding on others coat tails to avoid critical problems. The guy that sticks his neck out to gain these critical professional experiences, and handle them fluidly, is a core team member. Everyone else is a failure waiting to happen, or more simply a naive novice.
Sticking to these rules, will sometimes have unexpected costs. I joined Fortune Systems in Jan 1981 after pulling Onyx Systems out of the fire with a short schedule KISS UNIX V7 port to the Zilog Z8002. The Fortune Systems VP of Engineering I was working for was very much on the same page, and we lined up a small team of very seasoned developers to pull off a rushed/packed 9 month schedule to functional KISS prototypes by Comdex, and FCS a few months later. We were comfortable with this tight and demanding schedule … Marketing held back our staff funding until we put our feet solidy down with a “release engineering staff funding now, or slip schedules until you do” demand. We were replaced a couple days later by a pair of inexperienced “YES” men, who then promptly said yes to every new demand from marketing, hired several dozen young kids both in school or recent grads to “quickly” meet the schedule we threatened to slip. And the kids gleefully bartered schedule slips for new requirements until they were 18 months past initial FCS, and still 6 months or more from delivery. I switched hats, and started up a software team under the VP of operations, and kept my own Pert chart on my office wall that actually was pretty accurate in the end … within a couple months. Turned out the VP of Operations I worked for hated the BS that the new VP of Engineering floated daily, and protected me to add salt to the disruption they caused him. I was offered the Director of Software Engineer job again as things imploded, as they threatened to fire the three dozen kids. I declined while protecting the kids from getting axed. The company had just spent two years training a really good engineering team, tested by the worst of poor management fires. All they needed was a good manager to protect them from Marketing. The year plus delay, was greeted with strong sales and a $120M IPO. The Marketing guys were really good at their job, and they could gloss over their early staffing failure, and blame engineering.
Something don’t change … somethings do … what are best practice rules for 2020?
@TotallyLost – YES! Every bit of this! Every engineering manager should read every part of this. Thanks for sharing!
Thanks Kevin 🙂
We’ve been butting heads on this for a couple years now … it really is a management problem, often rooted in organizational culture issues, where developers are often the scapegoat. If you don’t fix it, the good developers (and managers) will ultimately migrate to the competition, rather than forcibly sending your worst developers (and managers) to the job pool (IE layoff or fire for cause) where they will end up at your competitors.
I deeply respect the sales and marketing side of the house, as they create our wonderful paychecks (and sometimes golden handcuffs). But unchecked, they will turn every sales lead (that rewards them with a large bonus/commission) into a critical development fire for features raised by sales prospects. Often that bonus/commission is far more important to them, than the organizational impact that changing development targets/features/schedules creates.
I do admit that sometimes the feature/product changes might be a good thing, especially if the bonus/commission was to be shared with the development staff actually responsible for delivering on the customer/prospects needs.