Pengutronix at Embedded World 2022

Welcome to our booth at the Embedded World 2022 in Nürnberg!

Finally, the developers, producers and users of embedded systems gather again at Embedded World in Nürnberg. Almost everything seems as usual, except that this year the trade fair is taking place in summer instead of winter. Our booth is set up and we are waiting for customers and visitors to talk about new developments in Embedded Linux systems, Open Source, updating strategies and our gorgeous experiences and inventions concerning remote working and testing.

For those unable to attend or those who didn't have enough time to see them all, here is a brief summary of both familiar (but updated) and entirely new demonstrators.

RAUC Field Update Demonstrator

Here we demonstrate robust and secure image-based updating with the flexible RAUC update framework. Six Raspberry Pis represent a fleet of devices that we would like to maintain.

Updating is always a necessary but critical step during the life cycle of an embedded system. Updates are important - either to fix system bugs, solve security problems or simply to add new features. But, as embedded devices are often placed in locations that make them difficult or expensive to access, updates must be performed unattended and thus have special needs in terms of robustness and security.

In practice, the actual deployment mechanism can vary from simple individual updates over USB sticks up to fully automated network-based deployments using sophisticated roll-out scheduling strategies.

RAUC itself focusses on being a lightweight service that reliably controls the process of updating your device with a new firmware version. It is flexible by design and can handle all common system layouts. And of course: it's Open Source and licensed under LGPL.

In our demo, all Raspberry Pis are connected via WiFi to a central deployment server running the Eclipse hawkBit roll-out framework. The RAUC hawkBit Client, also a RAUC project, interfaces between the hawkBit server and RAUC. This setup demonstrates one possibility of using RAUC in a larger context with custom roll-out scheduling.

The devices are running two variants of firmware: one based on Pengutronix' PTXdist, the other based on Yocto - two completely different Embedded Linux distribution build systems, but both equipped with RAUC.

In the demo, the update is scheduled in 3 phases. When the devices of the prior phase did update successfully, the next phase will be scheduled.

When an update is scheduled for a device, it will download the new image. After successful cryptographic verification, the image will be written to the next free slot. If this is successful, the new image will be marked as "active" and the system will reboot. The bootloader will now select the new image.

Latest developments and added features for RAUC are:

  • Streaming updates over HTTP(S) without the need for local intermediate bundle storage
  • Full encryption of update bundles
  • Built-in incremental updates for partition images

Labgrid / LXA Test Automation Controller

Engineering a working Linux for your Embedded Device is one thing. Maintaining this software stack over security updates and new features, maybe even for 10 or more years, is a completely different story.

Automated testing of the complete embedded software on real hardware plays an important role in maintaining the device for a long time. In this demonstration we will show you some tools that can help you with this challenge.

The first component is Labgrid: Labgrid is a Python-based Open Source board control library. It allows you to interactively control embedded devices on your desk - or at a remote location. It also has a strong focus on automation: You can either use it as a library in your scripts or as a pytest plugin. This way, you can run tests on real hardware that feel like pure software testing.

The next component is a product from our spin-off: Linux Automation. Our Test Automation Controller (TAC) connects the physical world to labgrid. It provides interfaces such as an UART, a power switch with voltage and current measurement, USB host and device ports and general purpose I/Os.

The test controller can be powered using Power over Ethernet (PoE) and has an integrated Ethernet switch. This means, that in most cases, you only have two connections to your test setup: power for your device under test and Power over Ethernet for the Test Automation Controller. The device runs Linux based on Yocto. Of course, the Yocto layers will be released as Open Source once the device is launched. It will most likely be available at the end of 2022.

In this demonstration we run a few tests on the device under test (DUT), in this case, a Beagle Bone with a separate motor board.

At the beginning of a test-run, labgrid switches the device off and provisions the newest software image on the SD-Card using the USB-SD-Mux. Afterwards, the DUT is powered on and labgrid interrupts the boot process to run a first test inside the bootloader. Then, the boot process is resumed and labgrid takes control over the Linux shell.

In the shell, the main functionality of this device is tested: We want to spin the disk at a given speed. By slowing the disk down manually, we can let the tests fail intentionally. In a real-world scenario these results would now be collected by the test runner and some engineer would have to investigate what has happened. But in our demonstrator the process simply starts over again.

i.MX8MQ Four Video Demo

Here, we will show you a use-case for hardware-accelerated video decoding and output on the NXP i.MX8MQ. The hardware is an entertainment panel, as sometimes seen on airplanes or long distance coaches and features a 4K display. The real-world use-case for this device would be to play videos either from local or network storage or from your own smartphone. The software stack running on this device consists of Mainline Linux, GStreamer, Qt Quick, OpenGL and the Etnaviv graphics driver. So, the software is one hundred percent Open Source - no vendor blobs needed. This gives you full control over the software running on your device and reduces dependencies on any single supplier.

In this blogpost, Lucas Stach takes a deep dive into the issues we had with the closed source driver for this GPU in the past and shows how the Open Source drivers solved these problems.

Rockchip with Mainline Linux

In this demonstration we show you the driver for the VOP2 display controller used on modern Rockchip SoCs starting with RK3566 / RK3568, which was recently merged into the Mainline Linux kernel with 5.18. Think about it as a hardware unit that transforms a rectangular piece of memory into signals for your display.

Traditionally, our customers have been using well established European or American CPU vendors for their industrial, aerospace or medical devices. But since a few years Asian vendors have been targeting these markets as well. Now add a global chip shortage for devices from western suppliers: This makes devices from the Asian market even more attractive; simply because you can buy them.

In this case a customer selected a Rockchip SoC and decided to invest in mainlining the still-missing components into the Mainline Kernel. So our colleague Sascha had the opportunity to get the drivers for these components ready for merging by reworking them according to the requirements of the upstream kernel.

Open Software on Open Gateware

Here we demonstrate Linux booting to a login prompt on a soft core RISC-V. While this might not look like much, there are many interesting parts under the hood:

The board is an ECPIX-5, which is built around Lattice's ECP5, a medium size FPGA. In the last few years, the open FPGA toolchain community has improved the support for this FPGA to the point where it's often easier and faster to use LiteX, Yosys and nextpnr to generate the bitstream from a high-level description of the SoC.

The ECP5 has no built-in CPU, so we have to include a CPU in the FPGA bitstream. As RISC-V is free and open architecture, of course there are also free and open FPGA IP cores available such as the multi-core vexRiscv from SpinalHDL, which was used in our demo.

We used Yocto to automate the full build process of the gateware and software from source, ensuring full reproducibility. As the upstream FPGA toolchain is still evolving quickly, this approach makes it possible to perform pin-point fixes without the risk of running into incompatible changes.

Further Readings

Showcase: Fail-Safe (OTA) Field Updating

Being able to robustly and securely update embedded systems and IoT devices in the field is a key requirement of every product today. The update framework RAUC is the basis for a modern and future-proof solution. In this showcase we present the basic principles of a fail-safe update system and how Pengutronix can support you with implement this for your platform.

Showcase: Graphics on i.MX8MP

Enabling the graphics output pipeline on the i.MX8M Plus (i.MX8MP for short), is the most recent example on how open-source and upstream driver support for GPU and display engines can reduce effort and risk in a new project.

Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.

Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.

Linux Automation Test Automation Controller: A one Device labgrid Exporter

Our subsidiary Linux Automation GmbH introduces the LXA TAC (Linux Automation Test Automation Controller): an all-in-one labgrid exporter. The LXA TAC offers the usual interfaces to control one or more embedded devices (DUTs, devices under test) interactively or automatically with labgrid.

umpf - Git on a New Level

Modern software development is commonly accompanied by a version control system like Git in order to have changes to the source code being documented in a traceable manner. Furthermore, any version state should be easily reproducible at any time. However, for work on complex projects such as the BSP ("Board Support Package") of an embedded system with its multiple development aspects, the simple stacking of the individual changes on top of each other does not scale.

Pulse Width Modulation (PWM) is easy, isn't it? - Turning it off and on again

Part of Uwe Kleine-König's work at Pengutronix is to review PWM (Pulse Width Modulation) drivers. In addition, he also sometimes refactors existing drivers and the Linux kernel PWM subsystem in general.

CLT-2022: Voll verteilt!

Unter dem Motto "Voll verteilt" finden die Chemnitzer Linux Tage auch 2022 im virtuellen Raum statt. Wie auch im letzten Jahr, könnt ihr uns in der bunten Pixelwelt des Workadventures treffen und auf einen Schnack über Linux, Open Source, oder neue Entwicklungen vorbei kommen.

labgrid Tutorials

This week, we started our series of YouTube labgrid tutorials. In the next few weeks we will publish more video tutorials showing you labgrid's features and giving you handy tips and tricks.

Update of our Remotelab equipment

If it looks like an advertising blogpost, reads like an advertising blogpost ... it probably is an advertising blogpost! Nobody likes to read advertisements and we don't like to write them at all, but like all proud parents, we would like to show you the new products that our corporate subsidiary, Linux Automation GmbH, has freshly added to their store. With these new products we, and maybe soon you, will complete (y)our Remotelab infrastructure.