The German put out his hand. “Do you have ze papers?”
Of course I had my papers. I’d been planning this for weeks. My papers were all in order, I’d practiced my rudimentary German, and I’d anticipated every question he might ask me, along with my answers. Just act cool, I told myself. Confident, but not over-confident. Just give the man what he wants and he’ll let you go on your way.
This man stood between me and freedom. But there was something about his manner that told me he wasn’t going to let me past his desk without a struggle. This could get ugly.
I looked at the papers neatly arranged on his desk. The sharpened pencils, the rubber stamps, the calipers. Over his shoulder, I looked out through the window of his small, government-approved office to the world outside. To the bright sunlight, the fresh air, and the sharp, craggy mountain peaks in the distance. At this time of year, the mountains of southern Germany should have a light dusting of snow, I knew. But these mountains were bare, almost brown, because we were in California’s Yosemite Valley.
I was facing off with Klaus Bichlmaier, an engineer from the TÜV, the Technischer Überwachungsverein (technical oversight association), one of many agencies scattered around the world that’s qualified to test and approve international safety specifications. Although TÜV is nominally a German organization, they have offices all over the globe, including this garden spot just outside the gates of Yosemite National Park. If you squint just right, it looks a bit like the Alps, but with hotter summers.
Klaus himself was a recent transplant, a 30-something engineer like myself, assigned to one of the California field offices to help test electronic gear. His job is to manage a group of other engineers who perform a dizzying array of tests involving EMI, safety, spark gaps, finger traps, drop tests, and who-knows-what-else. Turns out, Klaus was also a pretty fun guy. And his English was certainly better than my German.
The purpose of our meeting? To certify the safety of the industrial-robot controller that my colleagues and I had just spent the last year designing. Before it could go on sale, it had to be certified safe. Eight-foot-tall industrial robots can get pretty terrifyingly dangerous if there are bugs in the control code, the kinematics, or the failsafe mechanisms. Most governments quite sensibly require you to get an independent third-party safety certification before any such machine can be offered for sale. And today was the first day of our testing.
Naturally, there’s a lot of paperwork involved in certifying the safety of a giant robot. More than I expected, in fact, but Klaus and the other TÜV denizens had their ducks in a row. They’d done their homework before we even got there. Paperwork and engineering: two German stereotypes, on full display. I just hoped we’d pass some of the tests before we had to pack up all the equipment and haul it back to Silicon Valley.
If you think about it, certification testing requires a lot of creative thinking. Not what you might expect. After all, no safety standard can tell you exactly what to look for in a new piece of equipment, or precisely how to test it. Standards try to be helpful and specific, but they also try to allow for leeway and freedom of implementation. You don’t want every car, every refrigerator, or every robot controller to be exactly the same. So how do you guarantee their safety when each one is different? You get creative.
Klaus and his team got pretty creative with the relevant ISO and IEC specifications that applied to our machine. Can you stick a finger in it and shock yourself? Hmm, let’s see… They jabbed a rubber finger at the box and decided, no, an operator can’t hurt himself that way. Can you shut it off with one hand? Yes. Does it emit harmful noises or noxious fumes? Thankfully no.
The RFI/EMI testing was the hardest part, and that also explains why the TÜV office is located in the backwoods of eastern California. They wanted to get as far away from civilization as possible to get a “clean” radio environment. Then they prop your box up on a stand in the middle of an empty field and measure how much radio energy it gives off. Too much radiation at the wrong frequencies and you fail the test. Which we did, badly. Helpfully, Klaus handed us some metallic tape and suggested that we tape over some of the gaps in our metal box, pointing to the most likely offenders. Just by looking, he could predict where radio energy was sneaking out. He’s particularly suspicious of 1-mm gaps in our panels. The tape helps.
It took several hardware revisions – and several trips out to Yosemite – but we eventually passed all the mandatory tests and got to proudly display a TÜV sticker on the side of our robot. Good thing, or we all would have been out of a job. But the testing process gave me an entirely new appreciation of the process of testing. It’s not a cut-and-dried procedure, overseen by humorless drones who can’t get real engineering jobs. On the contrary, I was pleasantly surprised by how creative and inventive our tormentors were. Sure, they need to adhere to the letter of the specifications, many of which are vague or confusing. But they’re absolute masters at interpreting intent and suggesting solutions. Does this paragraph about saltwater intrusion apply to you? Nope, so let’s toss that one out. How about this section on 500-KV shock testing? Yes, so let’s think of a way to verify compliance. It’s actually kind of fun.
So it was with some anticipation that I met with the CEO of Verocel last week. Verocel is also a global certification company, although they focus on software testing and they’re not government-sponsored. This 100-person company checks to make sure your software meets all the relevant safety standards, which usually involves medical equipment, aeronautics, and transport controllers (think railroads and subway trains). Interestingly, Verocel does precious little testing of automotive software, for the startling reason that there aren’t really any automotive safety regulations to test. In their view, the automakers are woefully behind the curve in adopting any kind of standard for today’s in-car firmware, much less the self-driving cars of tomorrow. Last week’s total hack of a moving Jeep suggests that they’re probably right.
Nevertheless, Verocel does a brisk business verifying software and systems for areas that do have reliable safety-certification standards. And, in doing so, they’ve developed their own in-house tool that manages the process. Called VeroTrace, it’s a kind of CMS for testing, allowing a team of engineers to check-in and check-out portions of a client’s code. VeroTrace itself is independent of any particular standard; it works with DO-178B/C, EN50128, IEC 61508, or whatever. It manages the process of testing, not the tests themselves.
The upshot is that Verocel’s many engineers can each work on a different part of the overall software-certification project. Waiting on a bug fix from the client? No problem, just move on to a different part that’s ready to test. Testing one component to completion? Splendid; mark it as finished and move on. At every step, VeroTrace backs up the current status of the project so that it’s obsessively documented throughout. It’s like a make script for compliance testing.
Like TÜV, Verocel sometimes gets creative with testing and certifying compliance. The standards don’t always spell out exactly what and how to do things, so certification calls for some inventive thinking. In the end, the client gets a DVD (or more likely, a stack of them) with auditable traces of everything that happened and how it was all tested. It’s more than enough to document compliance. And you don’t have to travel to Yosemite, except maybe for a nice vacation in the mountains.
