Automated Testing Summit 2019

Jan, Rouven and Chris attended the Automated Testing Summit 2019 at the ELC-E in Lyon. The Summit began with Lightning Tasks on the progress in some projects. One highlight was Kevin Hilman's talk: "the bugs are too fast - and why we can't catch them."

He supported his statement with numbers from the syzbot project: The project currently finds about 3 bugs/day - and the project currently achieves only 7% code coverage. Using number of commits and the lines of code in a Linux release he estimated that we currently have up to 20.000 bugs in every release!

Real World Examples with Labgrid

After the lightning talks the Summit split up in two tracks. Pengutronix was able to contribute two presentations:

First Jan and Rouven showed a few Real-world Examples with Labgrid. Their presentation started with a short introduction into Labgrid's architecture. Afterwards they used the venue's WiFi to remote control am Embedded Linux Board connected to another board in the room to introduce Labgrid's remote infrastructure. They closed their presentation with demonstrations of test written in pytest and how Labgrid can be used to interface hardware like USB-Webcams, Power Supplies and oscilloscopes.

Unfortunately Jan and Rouven had to rush through their presentation after the projector caused a 15 minute delay.

Improving Embedded Testing

Afterwards Chris continued with his presentation New Ways Out of the Struggle of Testing Embedded Devices. In this presentation he showed the current structure of the Embedded Devices Lab at Pengutronix and what causes the most problems in this lab. He assumes that automated testing and interactive work at Pengutronix will increase over time.

He concluded that it is necessary to increase the reliability of the overall infrastructure to keep costs low and the satisfaction of his fellow developers high. Chris presented a solution where every Device under Test is controlled by an individual test controller - a design that has already been implemented by the MuxPi and the OpenTAC. This mostly aims add reducing the error domains and reduce the number of USB devices used in the lab.

Chris' slides are available in the eLinux Wiki.

Wrap Up

The summit was concluded by a panel discussion of Jan, Pawel and Tim. The participants on the summit were happy to identity some consensus in the community:

  • At last years ATS the participants agreed to use pdudaemon as a common layer to control power switches. On time for this year's ATS there was a new release. This release includes (besides other features) a direct (aka. daemonless) mode that allows other tools to use pdudaemon as a library without the need to have an additional daemon.
  • It seems like the testing community agreed on the Kernel CI Database (kcidb) as common way to store test results.
  • It also seems like the Linux Testing Project (LTP) Metadata-Format is a good candidate for a common way to describe test-metadata.
  • We agreed that we want to continue to use the automated-testing mailinglist provided by the Yocto Project.
  • Jan encouraged the participants to collect handy hardware used in board farms around the world in the eLinux Wiki.
  • Chris will continue to collect facts on how to design Embedded Devices in a way that automated testing of this device is possible. His documentation is available at Github. A readable version is available at ReadTheDocs.

The participants agreed to upload their slides to the eLinux Wiki - as of today the first PDFs are online.

Next Meetings

The last topic was how to continue the Automated Testing Summit. The participants agreed that a more hackfest-like approach would be best at the moment. It was suggested to meet at the Plumbers Conference (August 25-27, 2020, Halifax, Nova Scotia) for this purpose. Additionally there will be a Testing and Automation Track at FOSDEM 2020 in Brussels.

Weiterführende Links

Pengutronix auf der Embedded World 2024

Treffen Sie uns auf der Embedded World 2024 in Nürnberg! Sie finden uns wieder in Halle 4, Stand 4-261. Wie üblich zeigen wir auf unserem Messestand Demonstratoren zu aktuellen Themen.

FrOSCon 2023

In ein paar Stunden beginnt die 18. FrOSCon an der Hochschule Bonn-Rhein-Sieg. Pengutronix ist wieder mit einem kleinen Team vor Ort. An einem der Partner-Stände zeigen wir wieder einige unserer Aktivitäten in der Open Source Community. Dafür bringen wir unseren labgrid Demonstrator und die FPGA Demo mit.

DjangoCon Europe 2023

Django ist Pengutronix' Framework der Wahl für Software zur Abwicklung unserer Geschäftsprozesse. Diese internen Werkzeuge bieten zudem auch immer die Gelegenheit neuere Entwicklungen im Django-Universum auszuprobieren.