Are fully automated cars just around the corner? If you listen to breathless media stories, you’d probably believe so. But that may not be the case – not only could full automation be further away, but there may also be a long transition period before we get there – a time when, critically, there may be all kinds of cars on the roads that have different levels of automation.

Last week’s ICCAD conference featured a keynote by Nissan’s Maarten Sierhuis, addressing some of the realities of self-driving cars. His main point was to counter some of the press that suggests that full automation is imminent: it’s far harder than people realize. The discussion was illuminating, but it also raised for me a question: during this extended transition, if you borrow someone else’s car or, say, rent a car, how can you quickly assess what levels of automation are present in the unfamiliar vehicle?

In order to motivate the question, we need to dig into the vision that Mr. Sierhuis laid out.

An Automation Roadmap

This vision is one of graduated releases of automation. There are definite SAE-defined levels that specify such attributes as driver-in-charge, but assisted by technology (ADAS); car driving itself, with driver watching in case he/she needs to take over; and full automation, requiring no involvement by the… well, not driver anymore, but occupant/passenger.*

But more interesting were yet further gradations that he laid out, particularly in the partially automated regime.

• The first task that will be conquered – and much progress has been made on this – is freeway** driving. Here you have the luxury of no pedestrians or bicycles, and the lanes have nice visible lines demarcating exactly where the car should be. It’s a “keep it between the lines” exercise.

In such a car, once you leave the highway and join the surface streets, the driver needs to take back control and do some good old-fashioned driving.

• The next stage starts to address surface driving – and the big complication here are those pesky pedestrians and bicycles. So this phase starts by conquering main arterials, where there are few such distractions, making it easier to figure out what to do in any given situation.

Another off-freeway complication is the appearance of intersections – where lines disappear and cars sometimes have to negotiate who goes first (think either aggressive, like New York, or passive, like Portland). Four-way stops, in particular, are the devil, according to Mr. Sierhuis (albeit paraphrased). Then there’s road construction, which can change by the hour, making maps necessary but insufficient (even for a freeway). These are big challenges to solve – and presumably they’ll be solved first for big roads.

But enter into a residential area or a crowded urban center, and, again, the driver must take control back.

• Residential areas are tough because there are lots of parked cars, and there might be children playing or other human and pet activities that must be accounted for – a new order of difficulty, and yet another stage of automation.

But this level, once conquered, still requires the driver to take over when entering the urban center.

• “Main Street,” as he referred to it, is the toughest nut to crack. He showed videos of how pedestrians, bicycles, and cars interact in downtown Tehran and Amsterdam – exceedingly complex. And it relies on subtle reads of pedestrian body language to indicate intent and who goes while the other waits. Programming cars to know the laws is fine, but not sufficient, since many of the Dutch bicyclists and pedestrians were totally flouting the law.

This staging of automation goals implicitly incorporates yet another dimension: one that has “safe” as a first stop, then “annoying” as a next stop (it works, but not well), and you eventually get to “socially acceptable” – a notion that will vary dramatically even over short distances (city vs. suburbs vs. rural, for instance – not to mention country by country). An auto manufacturer must decide whether to improve a given level of automation first or proceed first to a next level of automation, cleaning it all up later.

This 2D set of gradations constitutes something of a roadmap for automotive automation – at least on the part of Nissan. Other companies might proceed down a different path. Which, we’ll see, is part of the problem.

Moving from Car to Car

This scenario motivates an important and, apparently, non-obvious question. It has to do with the long transitional period – 10 years? 20 years? More? – between now and full automation (if you accept that this endpoint will indeed come). I asked the question to Mr. Sierhuis in the Q&A session, but I can lay the question out more clearly here than I could when I knew that mine was going to be the last question and I needed to be brief.

Let’s start with the Nissan roadmap. Over time they will release cars that have incrementally more automation; think of these as automation tranches. Notably, cars from each tranche will remain on the road for many years, even as newer, more advanced models are launched. The first tranche might require full driver control at all times; a later one will have the cars handle the freeway on their own; a yet later one will have them handle arterials, etc. The speaker acknowledged that everyone will have to learn how their new car works.

But we don’t always drive only our own cars. We borrow our friends’ cars. We rent cars. Today, there’s a bit of getting used to a different car (especially if it’s a manual transmission, meaning you have to get the feel of each car’s clutch even if you’re an experienced driver). Or maybe you need to decipher a row of buttons with unrecognizable symbols indicating their purpose. Or maybe you have to learn the new touch-screen interface. But, realistically, the hardest thing is pulling up to the gas station and searching high and low for where the gas cap release lever is. Annoying, especially if you’re scrambling to make your flight, but certainly far from life-threatening.

Let’s move now into the future, when there’s this mix of cars on the road, coming from various automation tranches. Will this new car I’m driving for a short time require me to take over when exiting the freeway? When entering a residential area? When I reach downtown? Never? Getting that wrong could be really dangerous. So how am I going to know which level of automation a given car has?

Now let’s assume further that different car manufacturers follow different roadmaps, with different tranches. How do you get trained up in any given car to know what’s automated and what’s not? It can be a recipe for enormous confusion – not a good thing behind the wheel. It’s one thing if you have the time to train on your own new car. It’s quite something else to jump into a rental car, late for a meeting due to a delayed flight, and have to undergo lengthy training.

My specific question was whether work was underway to define standards that would help drivers to understand what the capabilities of any given car were. The somewhat concerning response to this was that standards for some kind of human-machine interface (HMI) – effectively, how the driver interacts with the car – will take a long time and will probably come at the expense of some lives. He noted how many lives it cost to get airplane safety to where it is today.

This is a fair point, but you’d also think we could learn from the airplane experience and take active early steps to mitigate lives lost. He did say that whatever is released should be safer than what came before – which is a nice principle, but is still pretty vague, especially when old failure modes are eliminated, but new ones introduced.

What seems clear to me is that we need some standards – early on. One might wish for standard roadmaps – so that all cars fall into one of a few well-defined, well-understood buckets. That would be simplest. But such standards also get push-back because they might limit implementation of fresh new ideas.

So, absent such a standard, we need a standard way of communicating what the automation level is in a given car. There could be a number of components to this solution. Having some way to get information when the car starts is one. Perhaps this information is displayed automatically with each start; perhaps there’s a button you can push to get it (since it might be annoying to see this on your own car every time). If the latter, then there needs to be a standard button with a standard symbol so that anyone can find it easily.

Any messages have to be consumable by anyone. Drivers may be young and inexperienced or old and used to older cars – and unable to read tiny text fast. They may be smart, or perhaps not so much. They may be engineers or they may be artists or anything else. In other words, a message stating, “This car adheres to SAE level 3 with residential handoff” in small text flashed on the display for 10 seconds will not work.

I did some Googling to see if there was anything addressing this, and I found that, just recently, the National Highway Traffic Safety Administration (NHTSA) released their Federal Automated Vehicles Policy. It seems to address many of the considerations required for an individual to drive a car (such as the hand-off from automated to manual driving), but, in my quick read, I didn’t notice anything that addresses the car-to-car short-term learning curve. In addition, for the moment, what it provides are guidelines, although it paves the way for possible future regulation. And, of course, it covers only the US.

My European colleague, Dick Sellwood, fortuitously at Electronica – and better connected automotively than am I – did some sanity checking and found no evidence that anyone is addressing this specific use case.

Granted, this talk was by one person from one company, and, to be clear, it was a good, detailed, illuminating talk. But it also illustrated what feels like a gap in coverage, which, at best, could lead to confusion; at worst…

If this issue is, in fact, being duly addressed elsewhere, that’s a good thing (I didn’t find it; please feel free to point it out in the comments if you know if or where this is happening). Assuming it isn’t, consider this a shout-out that we need standards that quickly make clear to every driver exactly what they’re driving, and we’ll need those standards earlier rather than later.

*The NHTSA lists the 5 SAE levels as follows:

Source: NHTSA document, “Federal Automated Vehicles Policy”

**The terms “freeway” and “highway” have different meanings in different places. On the East Coast, in particular, I’ve heard objections to referring to any toll road as a “freeway.” In California, there are very specific rules for defining a “freeway.” In some places, a “highway” might have limited access; in others, it might have intersections (perhaps spaced far apart). For the purposes of this piece, I use “freeway” and “highway” more or less interchangeably to refer to a limited access road, where pedestrians and bicycles are prohibited and where there is grade separation from surface streets. For those of you in the UK, think M.

More info:

Federal Automated Vehicle Policy (PDF)