In England there is a phrase used to describe a significant sub-class of embedded engineers – “Fred in a shed.” It is the guy who works for himself and carries out contract work for a range of different people. I prefer to think of him as a sub-set of “Harry the Hairy Hacker.” You must all know him – even when he is in part of a multi-national company and not in a shed. He has war stories of how he single-handedly saved a project by coding in machine code for 48 hours without a break, and then, with a piece of baling twine and a strip of duct tape, managed to keep the prototype hardware working long enough for a full debug. When he goes to sleep he counts sheep in hexadecimal, and he loves jokes about upgrading girlfriend 8.0 to wife 1.0.
Harry the Hairy Hacker is both the strength of a project and also its potential weakness. Simplifying somewhat, but after careful research I know this is not simplifying to the point of caricature, Harry hacks good code quickly, these days mainly in C or C++, throws it at the compiler and, when the compiler stops throwing back errors, hands it over. There is working code ready for implementation, normally fairly quickly. So what is the problem?
His code is often tight and terse, so runs well in memory-limited systems. Since he is a really clever hacker, Harry may often exploit quirks in the specific compiler or little-known features in the language. This produces good results. Or it does until integration starts throwing up problems, perhaps because the spec Harry was working to wasn’t the same as the one the hardware guys were using. Or until it is maintenance time, when someone in the field finds a problem. Or until there is a need for a variant of the product, defined by code, adding a few extra features that will sell well. Or until the decision is made to transfer to a different target, with a different compiler. Normally Harry doesn’t have to cope with these things: they are beneath him, or he is off with the next project, gone to another company or moved to another shed.
That’s when the terse code, often accompanied by equally terse documentation (if any) comes back to bite you. Lesser mortals need a guide through the maze: there are tools that can be that guide.
“The problem with tools is that they cost money.” Yes – indeed some of them do, and there are tools out there that can cost a great deal of money. But there are also tools that are free or available for a few hundred dollars. And what is the cost to the company of shipping faulty products? And the benefit to the company of reducing product development cycles?
“Tools take time to learn.” Yes – indeed they do, but there is significant evidence that, particularly after the first project, the time invested is amply repaid in faster projects with fewer errors.
“Tools restrict how I work. Programming is an art and I need to be free to express myself.” Perhaps, at the very highest level, programming is an art but there are very few people who can work continuously at the very highest level. For the rest of us, programming is probably more accurately described as a craft, where a set of skills can consistently produce competent and pleasing products. There are tools out there that can help harness these skills and make them work better, and we are going to look at tools for project definition, version management, coding standards and code checking.
Many people claim to have been the person who was told, “I am off to find out what it is we are working on. While I am out, you can start work coding it.” Good definition can be one of the toughest parts of any project. Back in the dark ages, when computers were kept in special rooms tended by an elite priesthood, it was a truism that the user was really able to define what he wanted only when the product was delivered. Now IT companies have a whole range of technologies to involve users in helping to define the end product. In embedded, the favourite tool for definition is modelling, normally using the Unified Modelling Language (UML). While UML tools are generally seen as big, expensive, and really for enterprise systems or safety-critical/high-reliability systems, there are tools out there that are none of these things. There are UML tools, complete with C or Java code generators for under $1000 dollars. Telelogic, whose Rhapsody and Tau tools are certainly not cheap, offers Telelogic Modeler as a free download. It allows you to develop models and use them to document the project definition. Obviously Telelogic is hoping that you will get hooked on it and then buy an upgrade to Rhapsody to ease your addiction and to run simulations and generate code.
Code generation is one of those areas that can cause steam to come out of Harry’s hairy ears. “No code generator can be as good as the best programmer!” Well possibly not, but programmers of that calibre are rare. The providers of code generating tools emphasise not only consistent code, but also speed. This speed is valuable not just at the first pass through the project, but when testing shows that there have to be changes to the definition. Some providers go even further. Esterel claims that its SCADE code generator (not a cheap product) produces output that is immediately qualifiable for DO-178B (up to Level A), certifiable for IEC 61508 certified at SIL 3 and EN 50128 certified at SIL 3/4. (Translating – that means defence and aerospace companies will accept that code created by SCADE is certifiably safe, although some people cast doubt on this. If you want more, a good place to look for debate is the safety critical mailing list, www.cs.york.ac.uk/hise/sc_list.php).
Another area that Harry usually ignores is version control. Why does he need it? He can remember what changes he made and when. Well, again mere mortals do need some form of version control, even if it is only a free download. Rolling back changes that didn’t do what you expected them to do or that generated issues with other parts of the system are just the starting point. If there is more than one of you working on a project, being able to keep track of who is using/changing what parts of the code can avoid longer-term issues. And when it comes to new implementations or different versions of an existing product, some form of version management is vital. Depending on the functionality and number of seats supported, version management tools can be frighteningly expensive, but even the top=end tools often have “community” versions for free download, hoping you will like it or need it so much that you will want to upgrade to the paid version.
Another steam generator is coding conventions, laying down how code should be laid out. For Harry, coding conventions and coding standards are an intrusion into his human rights. For ordinary mortals they serve to ensure compatibility between different programmers, to create readable code, and even to create code that will compile cleanly. There are lots of standards available, for example the Free Software foundation publishes a free C standard, www.gnu.org/prep/standards/ so you don’t even have to build a standard in-house. (Try searching C “coding standards” on Google – over one third of a million hits!)
One of a number of projects to tame the glorious jungle that is C (and C++) is driven by MISRA (Motor Industry Software Reliability Association). MISRA started life when automotive manufacturers began to see that the wide use of electronics in cars was going to require the equally wide use of software. Understandably, they were worried that unreliable software was likely to cost them money and even cost drivers and passengers their lives. MISRA C 2004 is the current version of a set of rules (currently 141) for using C to produce unambiguous code, and it has been used as the basis for many internal coding standards. It costs only £10 ($20) to download, and, even if you don’t follow the rules closely, they contain a lot of good sense. In the last couple of months MISRA has launched MISRA C++, which is in the same mould.
Having created code, then it is well worth running it through a few filters before compiling. Lint tools are cheap, some are even free, but even the cheapest one can pick up a ton of stuff that would otherwise sneak by the compiler and then lurk, waiting to cause problems in use. Static code analysis tools are more expensive but are capable of finding all sorts of nasties that are buried deep and that a compiler is not designed to catch.
There are MISRA C and MISRA C++ static code analysers from companies like LDRA and Programming Research, who were both on the MISRA C++ working group, that ensure that the code conforms to these rule sets. Figures from Alcatel show that static code analysis can reduce the project life cycle by as much as 50%, both by reducing software development time and by cutting back dramatically on the amount of debugging needed.
And this shows that it is this last family of tools that can have an immediate significant impact on a project, reducing debug time dramatically. Debugging can consume 50% or more of the project effort and time. Raw code, hot off the keyboard, can have 50-200 errors per hundred lines of code (according to Jack Ganssle – see below). If these bugs can be squashed before compiling, then there are many fewer problems to find at debug.
I had nearly finished writing this when I received a copy of the second edition of Jack Ganssle’s The Art of Designing Embedded Systems (Newnes 2008). It makes the points I have made and many more, with a far greater depth of technical support and detailed examples – of course, he does have 300 pages to work with. Jack is a great believer in formal code inspections, which I haven’t mentioned, and a great advocate of getting programmers out of cube farms and into private offices, which is well documented as a way of improving productivity greatly. (Given the enthusiasm that greeted Dilbert’s creator, Scott Adams, at the ESC a couple of years ago, most embedded developers would agree.) Some of Jack’s statistics have been retrofitted into this article.
Another statistic from Ganssle: 85% of companies won’t spend more than $1K on any but the most essential programming tools. So we can’t always blame Harry the Hairy Hacker for not using tools: the pointy headed manager, who keeps him in the cube farm, oftentimes won’t let him buy them.
Footnote: as well as the information from Jack Ganssle’s book and conversations with dozens of people, this article also owes a debt to Chris Hills, whose paper, Debugging is hard: so why do it was the trigger for finally getting something written.
