Embedded Linux Conference Europe 2023: Our Recommendations
Last month Pengutronix was present at the Embedded Open Source Summit (EOSS) in Prague. Thanks to all to all speakers for sharing your knowledge! In this blog post we want to shine a spotlight at a few talks that we found especially interesting. (Links to recordings will be added once the recordings are available.)
Chris' Suggestion: Testing and Remote Access to Embedded System DPI/LVDS Display Output (Marek Vasut)
In his talk Marek presented a way to capture the raw data of parallel display outputs and visualize the result. This allows easy debugging of the resulting output at the end of the complete chain.
The cool thing about his method is, that he does not need expensive or specialized hardware to do so. This solution incorporates the USB 3 version of a well-known parallel-to-USB buffered interface, that has been used with Sigrok as a logic analyzer for quite some time. He is using the Cypress FX3 USB controller together with the already existing fx3lafw Open Source firmware.
Marek used the FX3 dev-kit as a platform and added a interface PCB to his device under test (DUT). The interface mechanically adapts the connector on the DUT to the connector on the dev-kit and has no active parts. After some firmware modifications (fx3lafw samples free-running, but the parallel display signal has a synchronous clock signal) he was able to capture the raw data stream coming from the DUT. This also includes blanking areas, H-sync and V-sync signals.
In the end Marek was able to capture frames and visualize them.
A capture device for parallel video output was missing our toolkit for development and automated testing. It's great to see that all the building blocks are already there. I will try to build these setup myself once I am back at the office.
Roland's Suggestion: Accelerated Mainline Linux Development Ahead of SoC Availability (Bryan Brattlof & Praneeth Bajjuri)
Although their colleague Praneeth Bajjuri could not make it to the conference, Bryan Brattlof of Texas Instruments gave a good overview how the design and driver development process works for new SoCs.
They showed that the development process of a modern SoC is not at all straight-forward, and instead software and hardware development are heavily intertwined, which results in fast development cycles. With the different steps in the design process from the initial idea over hardware design and validation, to tapeout at the manufacturer, they are able to react to feedback by internal as well as external community members like business people, system developers and architects, and experts for the different Linux kernel subsystems. This way the SoC vendor can address past pain points and get a feeling whether a new SoC will be maintainable in mainline Linux at all.
Of course, the advantage of having kernel developers in-house makes it possible for TI to work with internal pre-release versions of new SoC designs, and develop preliminary driver support for new hardware units and the surrounding hardware ecosystem (e.g. PMICs) while the design is still being validated. Until hardware availability, the new systems are emulated in software based on Verilog and SystemC models, which is fairly slow (as Bryan put it, "You find other things to do after you hit that Run button"), but still, this approach allows them to find errors quite fast, and the turnaround for a bug fix can be measured in days or even hours rather than weeks.
In the Q&A, I was positively surprised that TI seems to have moved to a "upstream first" approach. When asked about how they persuaded the management, Bryan emphasised that they realised that the "downstream BSP" approach was not feasible for long-term platform support. They learned from that past mistake and decided to work with upstream directly instead, which gives them the advantage of feedback, support, and maintenance of the wider Kernel community. At last they want happy customers after all… :-)
Johannes' Suggestion: MUSE: MTD in Userspace, or How to Extend FUSE for Fun and Profit (Richard Weinberger)
NAND flash is prone to bits flipping now and then, which requires error correction to be implemented in NAND filesystems. Debugging with real-world NAND dumps can be quite painful, since looking at raw dumps in a hex editor is still somewhat state-of-the art.
In his talk Richard pointed out that the existing in-kernel NAND-flash simulators mtdram, block2mtd and nandsim besides qemu virt have been a good pathway for implementing and testing especially error correction code and for bringing the MTD framework in good shape, but they often do not simulate flash wear and are not very flexible with respect to fault injection and fuzzing.
When a colleague of Richard's mentioned cuse (character device in userspace) Richard was immediately hooked and thought about if cuse or the closely related fuse (filesystem in userspace) could be used for writing a new and more in-depth MTD simulator, for which he started writing MUSE (MTD in userspace). Richard soon found that he only needed to add some minor additions with under 1k LOC in the kernel to the already existing fuse implementation for simulating bad block handling and could quite easily plug MTD filesystems on top of a fuse MTD simulator.
He now aims to have full record/replay options for using realworld NAND dumps, recording single steps of error correction for better analysis, fuzzing, fault injection and supporting multiple formats of raw NAND dumps.
Richard still wants to experiment with his implementation, but announced that he will upstream his work once he figured out a stable set of required operations for the FUSE device in order to support the MTD simulation. He presented the internal structures and how FUSE works, as it uses a rather unique setup where the kernel is a client to the userspace server part with the kernel making requests which are answered by the userspace. At the Moment Richard is experimenting with using the libfuse low-level API in rust, while his original implementation has been written in C.
At the end of his talk, he also pointed out that this could possibly be (mis)used for userspace MTD drivers, though this is strongly discouraged, not only will this approach exhibit a performance penalty but also have some security implications is calling for lots of other havoc.
I really enjoyed this talk, as it shows a creative use of already existing infrastructure for solving real-world problems, I am curious if we will see similar FUSE-based approaches for tackling issues in other fields in our customer projects soon.
Michael's Suggestion: How Much Is Tracing? Measuring the Overhead Caused by the Tracing Infrastructure (Andreas Klinger)
Andreas reports on his investigation of the performance overhead of different tracing mechanisms. He uses a test setup consisting of a BeagleBone Black with custom Linux driver to toggle a GPIO and an oscilloscope to measure the pulse width of the GPIO. Enabling and configuring different tracing mechanisms in this test setup and observing the effects on the GPIO pulse width, allows to draw conclusions about the overhead of the different mechanisms.
While developing actual products, we already experienced that the tracing mechanisms, even though they are highly optimized and should have very little overhead if they are enabled at compile time, but disabled at runtime, don't come for free. Seeing the confirmation and actual measurements of this performance overhead is very insightful, as tracing is a very valuable debugging tool and one may prefer to keep it enabled in production systems.
Steven Rostedt, who is the maintainer of the tracing infrastructure, showed a lot of interest in the measurement results. This further underlines the relevance of Andreas' work. Hopefully, this work may result in further optimization of the tracing infrastructure or at least in some guidelines which tracing options may be kept enabled at what cost in production systems.
ELCE Booth Crawl
During the Onsite Attendee Reception the Embedded Linux (Micro-) Conference (ELCE) gives projects and companies the chance to present their projects with a tabletop demo. The booth crawl is always another good opportunity to get in touch with the community.
Penugtronix used this chance to present two of our tabletop demos:
Our Open Source FPGA Demo shows that with an Open Source toolchain, RISC-V soft-core, build system and some IP-cores it is possible to build complete systems.
Labgrid is a Python-based Open Source board control library. It allows you to interactively control embedded devices on your desk - or at a remote location. It also has a strong focus on automation: You can either use it as a library in your scripts or as a pytest plugin. This way, you can run tests on real hardware that feel like pure software testing.
The Test Automation Controller (TAC) (designed by our spin-off Linux Automation) connects the physical world to labgrid. It provides interfaces such as an UART, a power switch with voltage and current measurement, USB host and device ports and general purpose I/Os.
The test controller can be powered using Power over Ethernet (PoE) and has an integrated Ethernet switch. This means, that in most cases, you only have two connections to your test setup: power for your device under test and Power over Ethernet for the Test Automation Controller. The device runs Linux based on Yocto. The Yocto layer is already available under MIT license. The device will most likely be available in autumn 2023.
In this demonstration we run a few tests on the device under test (DUT), in this case, a Beagle Bone with a separate motor board. At the beginning of a test-run, labgrid switches the device off and provisions the newest software image on the SD-Card using the USB-SD-Mux. Afterwards, the DUT is powered on and labgrid interrupts the boot process to run a first test inside the bootloader. Then, the boot process is resumed and labgrid takes control over the Linux shell.
In the shell, the main functionality of this device is tested: We want to spin the disk at a given speed. By slowing the disk down manually, we can let the tests fail intentionally. In a real-world scenario these results would now be collected by the test runner and some engineer would have to investigate what has happened. But in our demonstrator the process simply starts over again.
This year the Embedded Linux Conference Europe (ELCE) is back in Prague! Pengutronix, again, is on a field trip with 15 colleges to attend the conference. The ELCE is one of the big conferences where the Embedded Linux Community meets during the year. This time the ELCE is part of the Embedded Open Source Summit (EOSS): a new conference with only embedded topics and without cloud- or crypto-tracks.
In a few hours, the 18th FrOSCon will begin at the Bonn-Rhein-Sieg University of Applied Sciences. Pengutronix will be there again with a small team. At one of the partner booths we will show some of our activities in the open source community. We will bring our labgrid demonstrator and the FPGA demo.
Django is Pengutronix' framework of choice for internal applications that handle our business processes. These internal tools are also a great opportunity to try out current developments in the Django universe.
After the Corona break, the Chemnitzer Linux-Tage will actully take place in real life again, and the Pengutronix team will have eight (!) talks there.
Friday, the last day of ELC-E 2022, is traditionally the day of the Embedded Linux Closing Game, with Tim Bird reporting about the embedded Linux world (universe?) domination progress and the overall state of the union. Of course, there were again several interesting talks.
The Convention Centre is directly at the water front of the Liffey river, just a few walking minutes away from O'Connell Bridge, Temple Bar and Trinity College. Visiting ELC-E is always a good opportunity to visit interesting cities in Europe. However, here is my story of the talks I heard on day 3.
The Dublin Convention Centre is huge - there is more than enough space for all the developers participating in the Open Source Summit. Fortunately, the talks will be on YouTube after the conference, so it's no problem that one can only hear a small selection of talks. However, here is my report of the talks I heard on the 2nd day of the conference.