Automated Testing Summit 2019

Jan, Rouven and Chris attended the Automated Testing Summit 2019 at the ELC-E in Lyon. The Summit began with Lightning Tasks on the progress in some projects. One highlight was Kevin Hilman's talk: "the bugs are too fast - and why we can't catch them."

He supported his statement with numbers from the syzbot project: The project currently finds about 3 bugs/day - and the project currently achieves only 7% code coverage. Using number of commits and the lines of code in a Linux release he estimated that we currently have up to 20.000 bugs in every release!

Real World Examples with Labgrid

After the lightning talks the Summit split up in two tracks. Pengutronix was able to contribute two presentations:

First Jan and Rouven showed a few Real-world Examples with Labgrid. Their presentation started with a short introduction into Labgrid's architecture. Afterwards they used the venue's WiFi to remote control am Embedded Linux Board connected to another board in the room to introduce Labgrid's remote infrastructure. They closed their presentation with demonstrations of test written in pytest and how Labgrid can be used to interface hardware like USB-Webcams, Power Supplies and oscilloscopes.

Unfortunately Jan and Rouven had to rush through their presentation after the projector caused a 15 minute delay.

Improving Embedded Testing

Afterwards Chris continued with his presentation New Ways Out of the Struggle of Testing Embedded Devices. In this presentation he showed the current structure of the Embedded Devices Lab at Pengutronix and what causes the most problems in this lab. He assumes that automated testing and interactive work at Pengutronix will increase over time.

He concluded that it is necessary to increase the reliability of the overall infrastructure to keep costs low and the satisfaction of his fellow developers high. Chris presented a solution where every Device under Test is controlled by an individual test controller - a design that has already been implemented by the MuxPi and the OpenTAC. This mostly aims add reducing the error domains and reduce the number of USB devices used in the lab.

Chris' slides are available in the eLinux Wiki.

Wrap Up

The summit was concluded by a panel discussion of Jan, Pawel and Tim. The participants on the summit were happy to identity some consensus in the community:

  • At last years ATS the participants agreed to use pdudaemon as a common layer to control power switches. On time for this year's ATS there was a new release. This release includes (besides other features) a direct (aka. daemonless) mode that allows other tools to use pdudaemon as a library without the need to have an additional daemon.
  • It seems like the testing community agreed on the Kernel CI Database (kcidb) as common way to store test results.
  • It also seems like the Linux Testing Project (LTP) Metadata-Format is a good candidate for a common way to describe test-metadata.
  • We agreed that we want to continue to use the automated-testing mailinglist provided by the Yocto Project.
  • Jan encouraged the participants to collect handy hardware used in board farms around the world in the eLinux Wiki.
  • Chris will continue to collect facts on how to design Embedded Devices in a way that automated testing of this device is possible. His documentation is available at Github. A readable version is available at ReadTheDocs.

The participants agreed to upload their slides to the eLinux Wiki - as of today the first PDFs are online.

Next Meetings

The last topic was how to continue the Automated Testing Summit. The participants agreed that a more hackfest-like approach would be best at the moment. It was suggested to meet at the Plumbers Conference (August 25-27, 2020, Halifax, Nova Scotia) for this purpose. Additionally there will be a Testing and Automation Track at FOSDEM 2020 in Brussels.


Further Readings

FrOSCon 2023

In a few hours, the 18th FrOSCon will begin at the Bonn-Rhein-Sieg University of Applied Sciences. Pengutronix will be there again with a small team. At one of the partner booths we will show some of our activities in the open source community. We will bring our labgrid demonstrator and the FPGA demo.


DjangoCon Europe 2023

Django is Pengutronix' framework of choice for internal applications that handle our business processes. These internal tools are also a great opportunity to try out current developments in the Django universe.


Chemnitzer Linux-Tage 2023

After the Corona break, the Chemnitzer Linux-Tage will actully take place in real life again, and the Pengutronix team will have eight (!) talks there.