“Don’t sweat the petty things and don’t pet the sweaty things.” – George Carlin
In economic terms, you always want to give the easiest jobs to the lowest-paid employees. Save the hard stuff for the expensive and highly trained specialists. Ideally, you want each person working on a task they’re just barely qualified to perform; anything else is a waste of their talents and the company’s money. You don’t hire brain surgeons to dig ditches.
That’s a pretty cold and hard-nosed approach, but it makes commercial sense. In the engineering lab, you give the new guy the easy job of soldering prototype boards, while the CTO in the plush corner office thinks important thoughts about architecture and innovation.
Trouble is, it’s often the low-level stuff that can make or break your product. How often has poor quality capsized an otherwise grand product idea? Bad execution sinks brilliant concepts. The receptionist can be the most important person in the building.
And it’s often the most obvious bugs that get overlooked. While we’re hunting for obscure transmission-line effects on the signal analyzer, the real problem might be a bad solder joint, a bent pin on a connector, or a missing wire. The dumb stuff gets overlooked because we’re all too busy – or too proud – to bother looking for them.
“It’s not the problems you look at; it’s the ones you ignore” that cause product re-spins, says Todd Westerhoff, marketing guy at Mentor Graphics. Too many engineering groups employ specialists (or groups of specialists) who examine only their own narrow area of expertise. You bring in the signal-integrity guy to bless the tricky high-speed stuff, but not the rest of the board. He’s too expensive and his talents are needed elsewhere. So… how do you verify the other 95% of the design?
“We eyeball it.” That’s the actual response Westerhoff hears from product-design groups all the time. They throw talent at the hard parts of the design, assuming – or perhaps just hoping – that the easy parts will take care of themselves.
Back when he had a real job in engineering, Westerhoff recalls how a new product was held up for almost year because the design team couldn’t locate the source of a tricky signal-integrity problem. Call in the experts from out of town! Fire up the high-end analyzers. Spin the expensive silicon design. Run another simulation. Pore over the EDA tools’ reports. Nothing worked.
“It turned out to be a devastatingly simple problem,” he admits. By looking at the package design as-built, anyone could see that there weren’t enough stitching vias between two ground planes. The signal-return path was terrible. It was a layout bug that the EDA tools shouldn’t have allowed. But it was clear as day, once you looked. Nobody had checked the “easy stuff.” They’d all jumped to the conclusion that the problem must be subtle, exotic, and expensive. And the longer their bug hunt dragged out, the more they convinced themselves that it must be obscenely arcane. “It was supposed to be too simple to go wrong.”
What’s the solution? Use more EDA tools, says Westerhoff. While that might seem like an obvious and self-serving answer, coming from an EDA vendor, he’s also quick to point out that EDA tools are often inappropriate or misused. “EDA vendors tend to focus on the bleeding-edge problems and assume that the technology will trickle down to the average engineer. That’s wrong. It doesn’t trickle down, because it doesn’t solve the right problems.”
What we need to do, Westerhoff suggests, is to limit the bleeding-edge tools – such as the latest signal-integrity checker, for example – to the narrowly focused problems for which they’re designed. Then, use existing (and much cheaper) design-rule checkers for everything else. Save the diagnostic MRI for the rare brain tumors and stick with the familiar thermometer and stethoscope for everyday stuff.
DRC tools are like spellcheckers. They won’t turn you into Shakespeare, but they’ll catch a lot of typos. Remember, kids, those wavy lines in your Word document are probably there for a good reason.
Further, Westerhoff believes that each organization has its own peculiar “holes,” or areas that the engineers consistently overlook. That’s just down to the mix of skills and experience. Someone who spent months debugging a problem with bypass capacitors is unlikely to make the same mistake again, but the whole team might overlook problems with ground planes if nobody’s been bitten by that particular bug before.
Maybe EDA tools are more like dental floss. We hear the same advice from the dentist, year after year, but always ignore it. Eventually, the big orthodontia bill comes due and we lament those lost opportunities to do the right thing. Maybe this will be the year.
