Automated Testing Summit 2019

Chris Fiege, Jan Lübbe | | Event

Jan, Rouven and Chris attended the Automated Testing Summit 2019 at the ELC-E in Lyon. The Summit began with Lightning Tasks on the progress in some projects. One highlight was Kevin Hilman's talk: "the bugs are too fast - and why we can't catch them."

He supported his statement with numbers from the syzbot project: The project currently finds about 3 bugs/day - and the project currently achieves only 7% code coverage. Using number of commits and the lines of code in a Linux release he estimated that we currently have up to 20.000 bugs in every release!

Real World Examples with Labgrid

After the lightning talks the Summit split up in two tracks. Pengutronix was able to contribute two presentations:

First Jan and Rouven showed a few Real-world Examples with Labgrid. Their presentation started with a short introduction into Labgrid's architecture. Afterwards they used the venue's WiFi to remote control am Embedded Linux Board connected to another board in the room to introduce Labgrid's remote infrastructure. They closed their presentation with demonstrations of test written in pytest and how Labgrid can be used to interface hardware like USB-Webcams, Power Supplies and oscilloscopes.

Unfortunately Jan and Rouven had to rush through their presentation after the projector caused a 15 minute delay.

Improving Embedded Testing

Afterwards Chris continued with his presentation New Ways Out of the Struggle of Testing Embedded Devices. In this presentation he showed the current structure of the Embedded Devices Lab at Pengutronix and what causes the most problems in this lab. He assumes that automated testing and interactive work at Pengutronix will increase over time.

He concluded that it is necessary to increase the reliability of the overall infrastructure to keep costs low and the satisfaction of his fellow developers high. Chris presented a solution where every Device under Test is controlled by an individual test controller - a design that has already been implemented by the MuxPi and the OpenTAC. This mostly aims add reducing the error domains and reduce the number of USB devices used in the lab.

Chris' slides are available in the eLinux Wiki.

Wrap Up

The summit was concluded by a panel discussion of Jan, Pawel and Tim. The participants on the summit were happy to identity some consensus in the community:

  • At last years ATS the participants agreed to use pdudaemon as a common layer to control power switches. On time for this year's ATS there was a new release. This release includes (besides other features) a direct (aka. daemonless) mode that allows other tools to use pdudaemon as a library without the need to have an additional daemon.
  • It seems like the testing community agreed on the Kernel CI Database (kcidb) as common way to store test results.
  • It also seems like the Linux Testing Project (LTP) Metadata-Format is a good candidate for a common way to describe test-metadata.
  • We agreed that we want to continue to use the automated-testing mailinglist provided by the Yocto Project.
  • Jan encouraged the participants to collect handy hardware used in board farms around the world in the eLinux Wiki.
  • Chris will continue to collect facts on how to design Embedded Devices in a way that automated testing of this device is possible. His documentation is available at Github. A readable version is available at ReadTheDocs.

The participants agreed to upload their slides to the eLinux Wiki - as of today the first PDFs are online.

Next Meetings

The last topic was how to continue the Automated Testing Summit. The participants agreed that a more hackfest-like approach would be best at the moment. It was suggested to meet at the Plumbers Conference (August 25-27, 2020, Halifax, Nova Scotia) for this purpose. Additionally there will be a Testing and Automation Track at FOSDEM 2020 in Brussels.

Further Readings

ELC-NA 2020 - A Virtual Experience

Enrico Jörns, Jan Lübbe | | Event

In 2020, things tend to be a bit different from what we had before. This is also true for this year's Embedded Linux Conference in North America. The need for keeping physical distance required the Linux Foundation to switch form a real conference to a fully virtual one.

Pengutronix at Embedded World 2020

Yesterday, Embedded World started, in normal times one of the largest trade shows for embedded development in Europe. While many exhibitors (and thus maybe also lots of visitors) have canceled their presence due to the coronavirus, we present our booth and our demo show cases as usual.

FOSDEM 2020 – Recommended Talks

Enrico Jörns, Chris Fiege | | Event

FOSDEM is one of the biggest Open Source community meetings in Europe and takes place in Brussels at the Université Libre de Bruxelles every year in February. For Pengutronix this is always a good chance to meet developers, discuss current topics and enjoy some Belgian beer and food. This year we attended FOSDEM with 15 colleagues. Here are some talks our colleagues recommend you to see.