rsc's Diary: ELC-E 2022 - Day 2

The Dublin Convention Centre is huge - there is more than enough space for all the developers participating in the Open Source Summit. Fortunately, the talks will be on YouTube after the conference, so it's no problem that one can only hear a small selection of talks. However, here is my report of the talks I heard on the 2nd day of the conference.

Board Farms for Everyone: Making Hardware Debugging Easier and Sharing Boards Across the Globe!

Open Source is about collaboration (often even between competitors working in the same market segment), and thus this year's labgrid test lab automation talk was not given by Pengutronix, but by Christopher Obbard (Collabora).

In the beginning, Chris motivated that it is often necessary to "try something out" on a board, but this board could be in some remote location. So flashing a remote board and automated testing is something the Collabora guys needed as much as we did. Chris explained that, in his local lab, he needs about 10 boards, the lab needs to be reconfigurable quickly, it has to be low power and silent (WAF!). His lab uses a Raspberry Pi 4 as a controller with USB serial converters for the console (with the same robustness issues we experienced) and a network switch. He uses ansible to set up the software on the Raspi.

On the software side, he started with LAVA, but quickly found that it is mainly made for automated test runs, but not for interactive work. He found the maintenance effort to be high; flashing full disk images is difficult, and the loop to run a job takes quite a long time. So for a home lab and interactive work with a focus on actual development, LAVA was not suitable.

For interactive work, controlling the power and turning boards on and off is important, as well as flashing and booting into the software. The same requirements lead us at Pengutronix to developing labgrid, as a python framework for hardware test automation. A controller maintains system state and is able to connect to several exporters that actually talk to the devices, abstracting things like power ports and serial interfaces. Chris found that the simplicity to configure labgrid is really a bonus: just write two YAML files with a hardware description and places and be done. For interactive use, you can lock and unlock a board, so it is possible to share a place with a CI. To switch 230 V devices, he uses Sonoff WiFi PDUs that can be controlled with MQTT. For automatic tests, Chris uses GitLab CI.

All in all, Chris seems to be quite happy with his lab. A small pain point is that there is no standard way to upload files to the exporter. However, it is good to know that people seem to be happy with our open source projects...

Using OpenEmbedded Inside Containers? How and Why?

The 2nd talk I heard was given by Drew Moseley from Toradex, who reported about his experiences with OpenEmbedded inside containers. According to his experience, containers are a convenient way to package software without dependencies to the host system and deliver it to the embedded system. Containers mainly make use of standard kernel functionality to encapsulate software; however, in contrast to virtual machines, there is no separate kernel. Drew outlined that, with container registries like Docker Hub, it is pretty easy to install any kind of software without struggles.

The main question is: how do you create containers? One common problem with predefined container registries is that you have to trust a container image made by someone else, and the usual distribution containers include a lot of standard software that might not even be needed. With OE, pretty small container images can be built - I leave the details out here, so if you are interested, the video of the talk should be available soon.

Leveraging Next Generation Cellular Networks for Drone Telemetry and Payload Communication

Unfortunately, the "How Automotive Grade Linux UCB Transitioned to Make Use of Flutter" talk was cancelled; the next talk by Ngonidzashe Mombeshora couldn't be held in person due to visa issues, but was shown as a prerecorded video.

The advantage of using mobile networks for drone control is that cellular networks are available almost everywhere. However, to be able to fly a drone in realtime, latency is pretty important, so in order to optimize for latency, he first built an SDR based open source 4G network for drone communication, based on an SDR hat for a Raspberry Pi and a set of docker containers for the software stack. Then, a 2nd generation testbed with 5G was set up and he achieved 90/30 MBit/s down-/uplink at 5 ms latency. Compared to commercial networks, the testbed had much better performance w.r.t. latency. Finally, he run a drone testflight in his own non-public mobile network, using MavLink for telemetry.

All in all it was an impressive talk, but of course mainly from an academic point of view.

Automating and Managing an IoT Fleet Using Git

In my last talk of the day, Matthias Lüscher (Schindler AG) talked about how to manage an IoT fleet with git. The mission of his open source project, edi, is to automate as much as possible for building, managing and delivering embedded Linux for embedded devices. The setup starts with Debian for the packages, he then constructs a Mender update image and deploys it to a test field first, runs his testsuite with Testinfra and gets test results back. If everything goes well, he gets a qualified image for the target device.

To avoid a golden image approach, Matthias uses debootstrap, runs the resulting minimal Debian image in QEMU and integrates the necessary changes and configs with the help of ansible. Testinfra turned out as a convenient environment to write tests in pytest, as it can be run on a development machine and work against an embedded target.

As an example, Matthias showed an edi workflow that turned a Raspberry Pi into a GitHub Action Runner, using the tools outlined above. In result, the runner registers itself at GitHub. In a 2nd example, a Raspberry Pi was turned into a Kiosk computer with 6 lines of ansible code.

In order to move from a single device to a whole fleet, Matthias showed his home demo fleet with devices from legacy Raspberry Pi 2 up to current i.MX8M evalkits. He models his fleet in a set of git branches: a development branch is where all of the development takes place. Then a main branch collects more complete features, and a canary branch is what is delivered to for example a realistic but nicely accessible environment. If nothing explodes there, things are rolled out into the real production environment. Clients usually work in pull mode, he uses Mender and the clients look if there is something new for them to pull in.

As devices might be offline at the moment of change, it is necessary to monitor the field setup, and for huge fleets, device definitions will probably not be put in an ansible inventory but into some database. However, small fleets work nicely with everything just living in git.

The toolbox turned out to be extremely powerful, and it is even possible to exchange the individual components, i.e. use Yocto instead of Debian etc.

Booth Crawl

Traditionally, there is a booth crawl where several projects show their embedded Linux demos. Unfortunately, this year there were only 6 demos, presented by three groups, one of which was Pengutronix. Hopefully there will be more in the next years, as the demos are always a highlight.

The Pengutronix crew demonstrated the labgrid test automation framework, the LXA Test Automation Controller (TAC) prototype, the RAUC update framework and our recent mainlining activities for Rockchip SoCs. Like every year, there have been lots of interesting talks with an interested audience.

Further Readings

rsc's Diary: ELC-E 2022 - Day 1

After two years with online conferences, the embedded Linux community once again meets in person for their annual Embedded Linux Conference Europe in Dublin, Ireland this year. Since many years, ELC-E is part of the Linux Foundation's Open Source Summit, and the best opportunity of the year to meet other Linux kernel and wider core ecosystem developers and discuss recent and future technological topics.

rsc's Diary: ELC-E 2022 - Day 2

The Dublin Convention Centre is huge - there is more than enough space for all the developers participating in the Open Source Summit. Fortunately, the talks will be on YouTube after the conference, so it's no problem that one can only hear a small selection of talks. However, here is my report of the talks I heard on the 2nd day of the conference.

rsc's Diary: ELC-E 2022 - Day 3

The Convention Centre is directly at the water front of the Liffey river, just a few walking minutes away from O'Connell Bridge, Temple Bar and Trinity College. Visiting ELC-E is always a good opportunity to visit interesting cities in Europe. However, here is my story of the talks I heard on day 3.

Pengutronix at Embedded World 2024

Meet Pengutronix at the Embedded World 2024 in Nurnberg! You find us, as always, in hall 4, booth 4-261. As usual, we will be showing demonstrators on current topics at our exhibition stand.

Chemnitzer Linux-Tage 2024

Pengutronix war auch in diesem Jahr wieder auf den Chemnitzer Linux Tagen dabei. Wie jedes Jahr sind die CLT eine willkommene Gelegenheit Freunde zu treffen und sich über Linux, Open Source und den Rest der Welt auszutauschen.

FrOSCon 2023

In a few hours, the 18th FrOSCon will begin at the Bonn-Rhein-Sieg University of Applied Sciences. Pengutronix will be there again with a small team. At one of the partner booths we will show some of our activities in the open source community. We will bring our labgrid demonstrator and the FPGA demo.

Embedded Linux Conference Europe 2023: Our Recommendations

Last month Pengutronix was present at the Embedded Open Source Summit (EOSS) in Prague. Thanks to all to all speakers for sharing your knowledge! In this blog post we want to shine a spotlight at a few talks that we found especially interesting. (Links to recordings will be added once the recordings are available.)

Embedded Linux Conference Europe 2023: Our Contributions

This year the Embedded Linux Conference Europe (ELCE) is back in Prague! Pengutronix, again, is on a field trip with 15 colleges to attend the conference. The ELCE is one of the big conferences where the Embedded Linux Community meets during the year. This time the ELCE is part of the Embedded Open Source Summit (EOSS): a new conference with only embedded topics and without cloud- or crypto-tracks.

rsc's Diary: ELC-E 2022 - Day 4

Friday, the last day of ELC-E 2022, is traditionally the day of the Embedded Linux Closing Game, with Tim Bird reporting about the embedded Linux world (universe?) domination progress and the overall state of the union. Of course, there were again several interesting talks.

Pengutronix on ELC 2021

The sun is shining for the last days of summer 2021. It doesn't only mean that autumn is coming, but also that this year's ELC is in preparation.