Streaming AI Development and Deployment at the Edge

I’ve said it before, and I’ll doubtless say it again—I like a simple story that I can wrap (what I laughingly call) my mind around. Too many times have I received a briefing from some overly enthusiastic company spokes-wiffle-waffle-person—you know the kind, they talk quickly and wave their arms around a lot—only for us to arrive at the end of the presentation with me finding myself wearing a rictus smile and thinking, “Say what?” to myself.

Bearing this in mind, you will be happy to hear that I just heard a story that is so simple in concept that even I can understand it. Of course, it probably goes without saying that the underlying technology is so complex and sophisticated as to make one’s eyes water, but the end-user perspective is as enticingly uncomplicated as one would dare hope.

So, here’s the deal… I was just chatting with Bill Eichen, who is VP of Business Development at DeGirum (see also the DeGirum Has the Best Edge AI Story Ever column I posted in January 2023, which is closing in on two years ago as I pen these words—eek!).

So, we already have a clue that we are talking about deploying artificial intelligence (AI)—of which we can consider machine learning (ML) and inferencing to be a subset—to applications residing on the edge, which is where the “internet rubber” meets and greets the “real-world road.” In a crunchy nutshell, the tale Bill told can be boiled down into three salient points as follows (all the rest is in the details):

Almost everyone wants to add AI to their embedded/edge applications and devices.
Very few people have the expertise required to add AI to their embedded/edge applications and devices.
DeGirum makes it easy to add AI to embedded/edge applications and devices.

Is that enough? What? You want more? Well, if you insist. Let’s start by reminding ourselves that the folks at DeGirum have been around for about seven or eight years, they currently have 30+ people and are growing, and they are solely focused on AI at the edge.

Basically, creating AI applications and deploying them at the edge is complex. Most people don’t understand AI all that well (although every year they understand it better). It’s expensive because there’s a lot of hardware engineers and software developers involved, and development times can be soooo looong.

Developing and deploying AI devices and applications is also time-consuming because when you take models from one platform to the next, you need to rewrite a lot of software. And last, but certainly not least, everything is very fragmented. In the last five to ten years, a bunch of companies have popped into existence to provide AI-related services (data collection, data labeling, model training, model porting…), but there are few applicable standards in the industry, which can make things… interesting.

Developing applications for edge AI is hard (Source: DeGirum)

So, how can the folks at DeGirum make everyone’s lives easier? Well, they’ve just announced the release of their hardware agnostic PySDK (“Python Software development Kit”) that has been designed from the ground up to streamline AI development at the edge. As the folks at DeGirum say, “PySDK provides a unified software stack that allows developers to add AI to any application with just two lines of code.”

Hmmm, even I can write two lines of code!

However, I fear I’m getting a little ahead of myself. Let’s wind things back a bit. Let’s start with the fact that the folks at DeGirum have a Model Zoo—that is, a collection of pre-trained AI/ML models that are readily available for use—containing thousands of models.

High-level classification of the Model Zoo (Source: DeGirum)

These models can be loosely bundled into 18 “buckets,” including things like people detection, face detection, age estimation, gender classification, emotion classification (happy, sad, angry…), and so forth.

These models can be used to address the AI needs of a vast range of application areas, from retail analytics and digital signage, to smart cities, offices, and homes, to construction, infrastructure, and industrial automation, to… we are limited only by our own imaginations.

Example application areas (Source: DeGirum)

Furthermore, the models can be used in isolation or in conjunction with each other in a mindboggling array of combinations and permutations. Also, you can BYOMs (“bring your own models”) to the party if you wish.

Take retail analytics, for example. Assume we are in a department store that already contains lots of cameras. The people detection model could be used to determine how groups of people flock around the store and where they congregate the most.

Meanwhile, a combination of face detection, age estimation, and gender classification models could be used to determine which demographic groups are focusing their attentions on which displays and which products.

Similarly, in the case of digital signage applications, face detection, age estimation, and gender classification models could be augmented with the emotion classification model to provide focused advertising as each person walks past the sign.

Let’s now turn our attention to the rest of DeGirum’s offerings. Let’s start with a high-level view of the software side of things as illustrated below.

A stack of stuff (Source: DeGirum)

In the bottom right we see the supported hardware, which includes application processors (APs) like Intel, AMD, Texas Instruments (TI), NXP, Rockchip, and Raspberry Pi, along with accelerators like NVIDIA GPUs, Hailo, MemryX, Google Edge TPU and BrainChip (see also my recent Taking the Size and Power of Extreme Edge AI/ML to the Extreme Minimum column about BrainChip).

Above the hardware we see some of the supported AI architectures / frameworks / software libraries (whatever you want to call them). These include offerings like OpenVINO, TensorRT, TensorFlow Lite, and ONNX Runtime.

Next up we have DeGirumJS, which is a JavaScript SDK designed to enable seamless integration into web and mobile applications. With DeGirumJS, developers can run AI models in the browser, bringing in real-time object detection, image classification, and more.

At the top of the pile, we find the user’s application code (called “Business Logic” in this diagram). And, sitting just under the user’s application code (conceptually speaking), we find the DeGirum PySDK, which we briefly touched on earlier.

As I previously mentioned, the PySDK supports a wide variety of hardware options, giving developers the flexibility to choose the best hardware without vendor lock-in. What I didn’t mention was that the PySDK includes a comprehensive inference application programming interface (API), which can be used to manage essential tasks like decoding, resizing, pre-processing, inferencing, and post-processing.

Pretty good so far, eh? And I haven’t even told you the good part yet! This is the DeGirum AI Hub (think “cloud”), which enables rapid AI prototyping without the need for hardware setup.

This hub includes multiple instantiations of all the previously noted hardware platforms. It also provides developers with access to 1000+ production-ready ML models to kickstart development. The hub also features a model porting toolchain that converts training checkpoints into deployable models across various hardware targets. The hub also enables edge devices to load and update models on demand, eliminating the need to bundle models within applications.

What this means to you and me is that we can create and test applications using models from DeGirum’s Model Zoo, checking the performance of different hardware platforms, all in the cloud, until we decide on the optimal cost/performance tradeoff.

When it comes to deploying our applications at the edge, in addition to running in the cloud, DeGirum’s PySDK also has a runtime version that runs in the edge device itself. In “marketing speak,” PySDK offers seamless transition from AI Hub to Edge, enabling developers to prototype in a hosted environment and then migrate to a run-time environment. Also, applications can be scaled easily by adding hardware without requiring changes to existing code.

“O-M-Gosh,” I hear you cry; “this is extremely exciting,” I hear you exclaim, shortly followed by your quizzical, “Just a moment, how much does all this cost?”

“Ah Ha!” I reply, it’s free! Well, let me qualify that a little. You can use the DeGirum’s AI Hub, PySDK, DeGirumJS, and models from the Model Zoo to develop and evaluate your applications for free. It’s only when you deploy your applications in the field that you are obliged to pay a license fee for the PySDK runtime. How much? I don’t have a clue! The money side of things bores me (unless I’m the one who must pay for it, of course).

So, what do you think about all this? As always, I delight in casting my orbs over your captivating comments and insightful questions.