2022 at Pengutronix

At Pengutronix and in the embedded Linux world in general, exciting things happen all year round, but a carry in the date field is a great opportunity to sit back and talk about it. In the broad categories of kernel, open source software, hardware and public relations we want to tell you what happened at Pengutronix in 2022.

Since without it, Pengutronix would only be an embedded … service provider instead of an embedded Linux service provider, we start with our contributions to the Linux kernel, especially to the PWM subsystem.

The category "State of the Union our Open Source Projects" is about the bootloader Barebox and the DistroKit BSP, which have learned to run Raspberry Pi 4 boards in 64Bit mode, and the PTXdist embedded distribution builder, which has been extended by many convenience features, e.g. to improve development in IDEs. Also in this category we are talking about the Lab automation software labgrid, which can now talk to even more automation hardware and is also being further developed architecturally. Last but definitely not least, we are happy to find our software updating software RAUC in various commercial products without our intervention, a good sign for a healthy open source project.

Also in the Pengutronix/Linux Automation GmbH hardware world there have been some developments, even though the chip-shortage still slows us down a lot. For example, our decision to mass produce and sell the open hardware project candleLight (a USB to CAN adapter with excellent Linux driver support) has been well received. But there has also been some progress with our own developments: USB to 10Base-T1L Ethernet Adapter and the USB-Mux are now available for purchase in the Linux Automation GmbH shop, and the LXA TAC is also moving step by step towards market maturity.

The review is rounded off by our podcast, launched at the end of the year, with scary stories from embedded hardware and software development. In two episodes so far, you can learn more about speculative execution in processors (and how it can go wrong) and other tricky bug cases than you (and we) ever wanted to know about.

Linux-Kernel

In the six kernel releases (v5.16 to v6.1) this year, we contributed more than 800 patches for various subsystems.

One change that made it into the kernel towards the end of the year should be of particular interest beyond the embedded community: USB support has been added to the Realtek RTW88 driver. This means that some new WiFi USB sticks are now supported by Linux.

However, it still needs a little patience, because the change will only be part of the upcoming v6.2 release of Linux.

Another useful change regarding USB has already made it into the v6.0 release: With the new disable sysfs interface individual ports of USB hubs can now be switched on and off in a controlled manner in the kernel. For hubs that support power switching, the power supply will be switched off. The latter was previously only isolated with the tool uhubctl possible, but without changing the port state in the kernel, leading to inconsistent states.

With a follow-up change we introduced, uhubctl now also uses the new sysfs kernel interface.

PWM

In 2022, maintenance in the PWM kernel subsystem was characterised by clean-up work and standardisation. A prominent feature is that the PWM framework now only supports the atomic updates abstraction for hardware drivers implemented since 2016, which can be used to make configuration changes with a single API call. This means that (with the appropriate implementation of the hardware driver and the appropriate hardware) unwanted voltage swings can be prevented when reconfiguring the PWM. In 2022, the legacy of converting all drivers to this model was finally tackled and completed with the change pwm: Drop support for legacy drivers.

One change that is not yet complete is that users of the PWM API also use the atomic API described above. This is one of the next goals.

State of the Union our Open Source Projects

Barebox

Barebox is also being diligently developed further and so there are some newly supported boards:

  • Quartz64 board
  • BananaPi R2 Pro
  • Radxa Rock3 board
  • 96Boards Meerkat96
  • Enclustra AA1
  • Raspberry Pi4 in 32bit and 64bit mode
  • PHYTEC phyCORE stm32mp1
  • InnoComm WB15 i.MX8MM
  • TQ mba8mpxl
  • TQMa6UL
  • MYIR MYD-AM335X

Features and improvements:

With improved TrustZone support, some newer SoCs can assign individual blocks of their hardware to either the Normal World or the Secure World. While the bootloader and Linux, for example, run in the Normal World, access to basic infrastructure such as clocks, resets and regulators can then only take place from the Secure World. Since drivers in the Normal World still have to communicate with parts of these hardware components, ARM introduced SCMI (System Control and Management Interface) as a standardised interface. The drivers in Barebox must therefore also be able to communicate with the Secure World, e.g. the ARM Trusted Firmware, via the corresponding Secure Montitor Calls, for which corresponding support was added in Barebox.

With the DSA (Distributed Switch Architecture) support in Barebox, it is now possible to connect a switch to the xMII interface of a SoC and thus support multiple network ports on a device even in the bootloader, for example for Netboot. The individual switch ports are represented as separate virtual network interfaces and can have separate network configurations set.

In order to optimally utilise the available network bandwidth for TFTP downloads, TFTP in Barebox now also supports the windowsize option from RFC7440. By carefully tuning the parameters, a speedup by a factor of 4-30 can be achieved for TFTP downloads.

Since Python2 has been discontinued for some time, the tools in Barebox are now completely based on Python3.

DistroKit

Users of our example BSP DistroKit could look forward to 90 patches in 2022. In addition to updates to the greatest and latest releases of various components, DistroKit now supports two more boards: the Raspberry Pi 4 and the Radxa Rock 3-A.

Something has also changed on existing boards: The new FIP boot format is now supported for the STM32MP1 boards and all ARMv7-A-compatible Raspberry Pi boards are now supported with the same image, regardless of whether Raspberry Pi 2, 3 or 4.

labgrid

In the embedded board control library labgrid, we have added resource and driver support for several new use cases this year, including for

Among a host of bug fixes and other features, labgrid's documentation has been revised and some code samples have since been CI tested. Under the bonnet, we have switched to pyproject.toml. A CI job for releases should make it easier for us to do more frequent releases in the future.

Next, the authentication between coordinator, exporter and client will be switched (the current authentication will remain supported for a while). This paves the way for us to split dependencies from the coordinator (currently Crossbar) and labgrid itself. The goal is to allow unpinned dependencies. Version conflicts of dependencies should thus become the exception.

PTXdist

In addition to the usual maintenance of all the packages, this year there have been improvements in the user experience.

With the new ptxdist subcommands fast-bsp-report and full-bsp-report, yaml files can be created that contain a lot of information about the BSP. Details about the packages such as version, licence and patches, but also the toolchain used and what images are created. This can then be used to generate a licence report, for example, with further tooling. And there is already a small tool that retrieves the necessary information from this so that you can develop applications for the BSP with VS code.

For many packages PTXdist now generates a compile_commands.json file. This can be used by language servers like clangd to find the right headers for example. This makes editing code with editors and IDEs that use a language server much more comfortable. The prerequisite for this is that the build system of the package has support for it. Currently, this includes all CMake and Meson based packages, the Linux kernel and Barebox.

Optionally, PTXdist can now add a so-called MiniDebugInfo section to all binaries. This is an extremely reduced form of the debug symbols that is just enough to resolve function names. This allows applications like systemd-coredump to create and log backtraces at runtime, even if the normal debug symbols are missing. Very practical to store more information in the log in case of errors in the field without having to archive large amounts of data in the form of coredumps.

RAUC

For the RAUC update framework managed by Pengutronix, the past year was marked by a series of interesting developments in the area of OTA updates.

Our maintainers, who drive the project forward with a lot of passion, are particularly proud that RAUC has in the meantime made it into many popular products, such as Valve's game console 'Steam Deck' or, as part of Linux4ICE, into the displays of the ICEs.

In 478 patches spread over 3 releases, based on the new 'verity' bundle format introduced in 2020, three basic features were implemented:

For the first time, RAUC supports the streaming of update bundles via HTTP(S) without external tools. This means that the need for a buffer for update artefacts on the target system is a thing of the past. Since even today transmission bandwidths are still often limited, expensive or unstable, we have laid the foundation for various methods of delta-like transmission with the concept of 'adaptive updates'. The best thing about it: thanks to RAUC's ability to have random access to the bundle, this is done largely transparently. With 'block-hash-index', a first method for chunking and differential updates of block device images was created.

Since updates can also contain sensitive data or applications and both the transport routes and the storage locations for OTA updates cannot always be completely sealed off, there has often been a wish to be able to transmit encrypted update bundles in RAUC. This year, this point was also implemented as part of a project. With crypt bundles updates can now be fully encrypted for a large number of individual recipient systems, while it remains possible to use them for streaming and "adaptive updates".

New Hardware Developments

Together with our partner company Linux Automation GmbH, we have expanded our hardware toolbox, even though development and especially production require a lot of patience in times of scarce chips and electronic components.

candleLights

We are a bit surprised by the success of Leonard's video "CAN Protocol Reverse Engineering with Wireshark" and are pleased to receive requests from the community about variants and improvements relating to the Open Hardware USB-to-CAN adapter candleLight used there.

For better handling for example, Chris Fiege has designed a 3D printed case.

LXA TAC

Those who met us at a trade fair or conference this year may already have caught a glimpse of the LXA TAC, the Linux Automation GmbH Test Automation Controller. There will also be opportunities to do so at future trade fairs. However, if you want to catch more than just a glimpse, you will have to be patient for a while, as the problems with procuring components, which have existed since 2020, will mean that series production will be postponed for a while.

The waiting time until production gives us the opportunity to fine-tune and polish the software side. So future buyers have a lot to look forward to:

  • Clean Labgrid integration of the various features.
  • An open and easily customisable Yocto layer as a basis for your own operating system images, including instructions on how to e.g. deliver your own configuration files directly in the image. (Of course, there will also be official pre-built image.)
  • A system management daemon that makes the device controllable via web interface and API.

USB Mux

For a long time it was quiet about our USB Muxes, but as the saying goes, "What's a long time coming is finally becoming good!" In December 2022, we received the first devices from production and were able to pass them on to customers.

With the USB Mux, we are expanding our possibilities in test automation to USB interfaces. More detailed information is available in the data sheet.

The production of further devices will follow, so that we will see the USB-Mux in use next year.

USB-T1L - Adapter

The new 10Base-T1L Ethernet standard allows transmission rates of 10MBit/s via a single wire pair. Those who now feel reminded of the 1980s and the time of vampire terminals can rest assured: 10Base-T1L brings some improvements compared to the technology of yesteryear. 10Base-T1L allows point-to-point Ethernet connections of devices over distances of up to 1.5km-2km with low power requirements.

10Base-T1L is well on its way to becoming the industrial Ethernet standard for long distances.

The USB-T1L adapter is a tool for embedded developers who want to develop 10Base-T1l based products both in a timely manner and as close as possible to the actual product. For this purpose, the USB-T1L is designed as a firmware-less device and the Linux kernel of the connected device itself takes over the control and configuration of the 10Base-T1L Phy installed on the device.

A special feature of the USB-T1L is that it is our first in-house produced device for external sale. The USB-T1L is the device with which we inaugurated our new Pick & Place machines. This purchase will enable us to produce prototypes faster in the future and speed up our development cycles. Fingers crossed that component availability will also improve soon.

Podcast

On the way to multimedia world domination, we can now report another milestone in addition to our blog, microblogging and our YouTube channel: our own podcast!

The podcast "Kernel Panic - Board on Fire" (a reference to the dreaded Linux "kernel panic", single board computers and the infamous error message "lp0 on fire") is about telling campfire stories from the world of embedded software and hardware.

These campfire stories are, for example, about months of troubleshooting through the various subsystems of a device to finally find the right "working bit". Since the code changes from these searches are often inconspicuous one-liners, we want to highlight the search process with all its twists and turns in the podcast.

Episode 1 - 2022-11-24 "Speculative Execution and an Execute Never Bit":
 Ahmad Fatoum tells a story that starts with trying to turn on a device's backlight while it's still in the bootloader, then gets lost in the idiosyncrasies of executing programs on processors for a few months, and finally results in a handful of lines of code.
Episode 2 - 2022-12-24 "Watchdogs and (Not So) Ideal Components":
 Leonard Göhrs tells about a device that restarts now and then, and how he found the various reasons that are not the cause of this problem. It's about watchdogs, overtemperature, less than ideal electrical components and also a bit about reverse engineering on the hardware level.

Unfortunately, this podcast is only available in German language.

Send Patches

Sources

The motto of our Christmas greeting this year was "Send Patches". For this, Anna from Atelier *Kleinerwolkendrache* created a wonderful patch. In our open-source spirit we want to share the sources with you.

The entire Pengutronix team wishes all customers, contributors, hackers and friends of the Linux world domination a good and successful start into 2022!..

... and as few off-by-one errors as possible!