Are you familiar with the Tracealyzer and DevAlert tools from Percepio? How about the Luos distributed (not exactly an) operating system from Luos? I don’t know about you, but for me, the past few days have been jam-packed with my learning all sorts of exciting nuggets of knowledge and tidbits of trivia pertaining to the implementing, monitoring, and debugging of embedded systems using these little scamps.

I’m not too sure where to commence, so let’s all take a deep breath, start at the beginning, wend our way through the middle, and — hopefully — arrive at the end in one piece while still maintaining our sanity (and our attire).

The funny thing about the tools I’m poised to talk about is that (a) they are outrageously awesome while (b) they have somehow managed to “dodge under the radar” and remain relatively unknown to the vast majority of embedded systems developers.

Now, before we plunge headfirst into the fray with gusto and abandon (and, of course, aplomb), let’s once again remind ourselves that I’m a hardware design engineer by trade — what little code I condescend to capture would make a professional embedded systems software developer burst into tears if they were to see it and — it probably goes without saying (but I’ll say it anyway) — these would not be tears of joy.

My programs are rudimentary in nature, which makes it all the more embarrassing that they so rarely work as planned. As a result, I spend more time than you might imagine trying to work out what’s gone wrong and, occasionally, fixing it. Furthermore, I don’t have access to sophisticated software tools and I wouldn’t know what to do with them if I did, so the way I debug my programs involves a mishmash of flashing LEDs and sprinkling “Print” statements through my code-like confetti.

On top of all this, my programs typically run “bare metal,” which means they are executed directly on the microcontroller with no layers of operating system abstraction getting between me and my input/output (I/O) pins. In particular, I’ve never used a real-time-operating system (RTOS), although I flatter myself that I know enough to be dangerous due to my reading the books written by Jean Labrosse who founded Micrium (which is now part of Silicon Labs) and created the µC/OS, µC/OS-II, and µC/OS-III RTOSes (which were offered by Silicon Labs, but which were spun out in 2020 and are now open-source).

The core concept underlying an RTOS is that an application is formed from a collection of tasks. The RTOS acts like a conductor, prioritizing and rapidly switching back and forth between the various tasks, thereby giving the impression that multiple tasks are being executed at the same time on a single processor core. Things get even more exciting if the processor core supports multiple threads and/or the processor is comprised of multiple cores.

I can’t even imagine how complex it would be trying to debug an RTOS-based application of this ilk with tasks firing off all over the place, tasks that are currently running being suspended while other — higher-priority — tasks take over, and everything being affected by the whim of random events that are occurring asynchronously in the real world.

Percepio: Tracealyzer and DevAlert

Talking to my embedded software developer friends… well, acquaintances… well, people whose email addresses I have acquired by one means or another and who haven’t yet discovered how to block me, they say that using traditional software debugging tools on an RTOS-based system is akin to me, as a hardware designer, attempting to debug a complex piece of electronic equipment using a single-channel oscilloscope or a solo-channel logic analyzer.

All of which brings us to Percepio, whose flagship product is Tracealyzer. In hardware design terms I can wrap my brain around, this is like having access to the most sophisticated high-performance oscilloscope/logic analyzer combo on the planet, but one that has been created in such a way as to be intuitive and easy to use. In a nutshell, Tracealyzer allows you to observe everything that’s happening in your RTOS-controlled system, including which tasks are being executed when and who called who, all coupled with myriad views into the inner machinations of the system like CPU resource utilization and suchlike.

Tracealayzer provides multiple views into the operation of your RTOS-based application (Image source: Percepio)

In fact, you can even link to, and display, events in the real world like sensor readings causing tasks to fire and so forth. Take a look at this 4-minute video to see a brief introduction to Tracealyzer (if this short introduction whets your appetite, you might also enjoy this 15-minute version).

As an aside, while the chaps and chappesses at Percepio were giving me a live demo of Tracealyzer, it stirred memories of something similar I’d seen a couple of years ago when Jean Labrosse was showing off the latest and greatest capabilities provided by Micrium’s µC/OS-II and µC/OS-III environment. When I mentioned this to the folks at Percepio, they grinned and said, “That was our Tracealyzer.”

Now, as we all know to our cost, it’s one thing to be debugging a prototype system in the lab, but it’s quite a different “kettle of fish” to try to diagnose problems with an Internet of Things (IoT) device that has been deployed into the field. In order to address this problem, the guys and gals at Percepio introduced DevAlert.

DevAlert allows the developers of IoT devices to be informed of “situations of interest” associated with devices that have been deployed into the field (Image source: Percepio)

DevAlert is a cloud-based monitoring service that makes it easy for the developers of the device to be informed within seconds as to any “situations of interest,” such as errors, warnings, or timeout conditions. The provided diagnostics includes a compact software trace containing the timeline of software events just prior to the issue being detected and reported for use with Tracealyzer. This software trace provides the context of the issue, which is often crucial for understanding and fixing the problem.

Take a look at this 40-second video to see a brief introduction to DevAlert (if this short introduction tickles your fancy, you might also be interested in this 19-minute extravaganza).

If you are excited and eager to learn more about all of this, Dr. Johan Kraft, CEO of Percepio, will be giving a presentation on Painless Multithreading: How to Verify RTOS Best Practices at Runtime at the forthcoming 2021 Embedded Online Conference (EOC), which will be held May 17-18-19 & 20.

Also, embedded legend Jack Ganssle will be giving a must-see keynote, The Microprocessor at 50. Furthermore, by some strange quirk of fate, your humble narrator will also be giving a presentation titled Not your Grandmother’s Embedded Systems (see also my column Why Attend the Embedded Online Conference? for details about a special promo code to attend this virtual event).

Luos: Luos

Just when I thought things couldn’t get better… they did. My chum Ben Cook is the Founder and Director at Airspeed Electronics Ltd., which is an electronic design consultancy that’s based in the UK specializing in high-performance acoustic detection and tracking technology for counter-unmanned aircraft system (UAS) applications (try saying all of that without drawing a breath).

A couple of days ago, Ben pointed me at a company called Luos. I must admit, I was so excited by what I saw in this video that I immediately wrote a blog about it, Luos Open-Source Real-Time Orchestrator, because that’s what I do.

Following the posting of my blog, it wasn’t long before I heard from Nicolas Rabault and Emanuel Allely, who are CEO and COO at Luos, respectively. I’m currently up to my armpits in alligators fighting fires without a paddle (I never metaphor I didn’t like), so we tentatively set up a plan to have a video chat in a week or so’s time. But then I received a message from my old chum James Langbridge. The last I heard, James was lecturing on the IoT at some university in France, so you can only imagine my surprise and delight to discover that he was so excited by Luos’s technology that he’s accepted a position as their Technical Content Manager.

James and I had a long video conference earlier today as I pen these words, and now I’m excited by Luos technology too. Once again, remember that I’m not a software guy, so please excuse me if I mess things up with regard to my explanations below.

First of all, Luos is open-source, so you can get everything for free from the Luos GitHub page, after which you can use it to your heart’s content for a whopping 0€ per month, the only catch being that you get no support. Where the folks at Luos make their money is on support (see support pricing) and training and design services.

“But what actually is Luos?” I hear you cry. Well, I’m glad you asked. In a nutshell, Luos is a little like an RTOS while, at the same time, Luos is nothing like an RTOS (you can see what a good job James did of explaining things me). When we got down to the nitty-gritty, James revealed that “Everything in Luos is a service” and “A service is anything you want it to be” (“Ah Ha!” I thought, “Now we’re getting somewhere”).

Sadly, I fear I may not be elucidating, explicating, and expounding this quite as well as can James (happily, I’m sure he’d be delighted to clarify things in excruciating detail if you were to reach out to him at james.langbridge@luos.io). In the meantime, let’s come at this from another angle. There are two types of services. One type is tied to the hardware (i.e., to a specific microcontroller node), like a service that controls a LED that’s connected to one of that microcontroller’s I/O pins. The other type of service is purely software, which means it can run on any of the nodes (microcontrollers) in the network.

Suppose we create a service to control a LED and we set that service running on a node (i.e., a microcontroller). We can think of this service as a mini-program or a process. A master control loop running on that node will cycle round “tickling” this service along with any other services on the node. In the case of this particular service, it won’t do anything by itself until some higher-level application — or another service — calls it to say, “Turn your LED on” or “Turn your LED off.” Similarly, you could have services that read the values from switches and sensors, and services that… well, do anything really.

Meet the Luos open-source real-time orchestrator (Image source: Luos)

All of the nodes are networked together using RS-485 (or RS-232 or any industrial bus that brings a glow to your cheeks and a smile to your face). It doesn’t matter in what order you connect the nodes. On power-up, they inform each other as to the services they have at their disposal. Also, the nodes are hot-swappable.

In addition to one or a bunch of services, a node may also be running a higher-level application (think of this as a super-service) that organizes things and calls services like a conductor controlling an orchestra. Also, there may be multiple different higher-level applications running on the same node or on different nodes. As James told me, we no longer have a typical embedded system where one thing is in charge and if something goes wrong everything grinds to a halt. Instead, a Luos system is fault-tolerant because even if one node (or one application or service on that node) has problems, the other nodes and applications and services will happily keep on doing their thing.

One of the examples James presented to me was that of a service that accesses the values from a 3-axis accelerometer. In this case, any other part of the system can request these values, at which time the service will access the accelerometer, pre-process the data, and return the appropriate X/Y/Z values. At some stage in the future, if that particular accelerometer device goes end-of-life (EOL) or is superseded by a different model (newer, better, cheaper, more accurate…), a new service can be created with the same name as the original service, which means the new sensor can be deployed without disturbing the rest of the system.

There’s also something called the “Gate” (see the node with the Bluetooth and USB symbols in the image above), which provides a gateway between Luos and the outside world. This allows you to have Python running on a host computer (Windows, Mac, Linux…) sending and receiving serialized data to and from the Luos network. In turn, this leads us to the fact that you could create a higher-level application (again, think of this as a super-service) in Python running on your host machine. I think it was at this point that my brains started to leak out of my ears.

I fear we’ve reached (perhaps crossed) the bounds of my software expertise. However, while we are here, teetering on the brink of the abyss with one foot hanging over the edge and the other foot on a bar of wet soap, as it were, may I make so bold as to suggest that Luos may offer a new paradigm that’s well suited for our embedded systems future. I’m not saying that traditional RTOS-based embedded systems will all soon be swept away, but it seems to me that Luos provides some very attractive capabilities and functionalities that fit very well with modern design teams, projects, practices, and requirements.

As one final thought — a thought I’ve already shared with the guys and gals at Percepio and the chaps and chapesses at Luos — I think that the combination of Percepio’s Tracealyzer and Luos’s Luos could be something of a game-changer.

Eeek! Here’s a final, final thought — a postscript, if you will — because (literally seconds before going to press), James contacted me to inform me of a really good description of Luos that’s just been posted here. 

So, how about you? Do you have any thoughts you’d care to share on any of this?