Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.

How can we support you?

By using Continuous Testing (CT) on real hardware, we are able to detect and fix occurring problems at an early stage. Our development projects essentially consist of two phases:

Initial Product Development

Many projects start with an initial development phase: In this phase we support you in commissioning your hardware, as well as building a Board Support Package (BSP), which is exactly tailored to your hardware. In this phase we develop for example missing drivers, add missing functions to software components or optimize the boot time of your device. In this phase software components are usually constantly updated to the latest versions. At the same time we offer to develop tests for your hardware and your BSP. These tests check the core functions of the system during development. This phase usually ends with the release of your product.

Long Term Maintenance

In addition, we offer you a permanent maintenance of your BSP: We regularly (e.g. several times a year) update your system and ensure with the tests that the functions are still given.

The subsequent maintenance provides you with significant advantages: If a software component should cause an error in your product or a security gap should occur, you will always have an up-to-date system at your disposal. This enables you to have short release cycles. At the same time, the changes from update to update are smaller, so that the actual effort for maintenance is lower.

Please do not hesitate to contact us: In combination with your product knowlegdge, we will come up with a development and maintenance concept for your operating system components that is tailored to your requirements.

Real World Example

In the following example, a customer relies on Verified Boot for its product. A chain of cryptographic operations ensures that only an operating system signed by the manufacturer can be run on a device. Each component (e.g. the boot loader) checks whether the next software component to be loaded (e.g. the Linux kernel) has a valid signature. Only if this is the case, the respective next component is executed.

Further Information on Verified Boot

Our colleague Marc Kleine-Budde gave the overview talk "Verified Boot: From ROM to Userspace" at ELC Europe 2016. (Youtube, Slides)

The Verified Boot chain, from the configuration of the ROM code of the processor, over the boot loader, the Linux kernel up to the user space, were integrated into the BSP and are still being maintained by Pengutronix.

Test Case

To ensure that the verification of both the FIT image (including Linux kernel and device tree) and the dm verity protected file system work, they are tested. To do this, the FIT image is modified and an attempt is made to boot the system. In a different test, the file being executed as the init process in the dm verity protected file system is modified. Reading and executing this file is expected to fail, since the dm verity hash tree is corrupt. In both scenarios, the boot attempt fails, thus Verified Boot works as expected.

During this test, various functions of the barebox boot loader are tested as a side effect: Images are copied between partitions, file systems are mounted and modified and the triggering of the watchdog is tested.

Error Case

During a maintenance update of the BSP, the boot loader has now been updated and the previously discussed test cases failed. The good news: Verified Boot still worked: The modified system was not executed.

But: The test revealed two bugs in barebox: One bug occurred when copying to POSIX device-files (Fix by Sascha Hauer). The other bug concerns the watchdog (Fix by Ahmad Fatoum). It is expected to reset the system after the Verified Boot chain was corrupted.

Both errors were uncovered by the test suite and the appropriate fixes integrated into the BSP before the new BSP version was delivered to the customer.

Back to technical showcases

Further Readings

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.


Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.


Pengutronix at FOSDEM 2021

"FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. In 2021, they will gather online." -- FOSDEM


Pengutronix at Live Embedded Event

conference, event, testing

Now that, due to the COVID-19 pandemic, everyone has gotten used to digitalisation and online conferences - it has never been easier to organise a conference and bring together all experts and interested parties for a few hours of intensive exchange of ideas on a certain topic.


#FlattenTheCurve – Introducing Our Remote Setup

The Corona crisis is a challenge that has hit many people as well as most companies quite unexpectedly. The entire team of Pengutronix wants to thank those that currently ensure our essential supplies, health system and civil infrastructure!


A Logo for labgrid

Enrico Jörns | | labgrid

It took us a while to find a good logo for one of our latest (but already quite-a-few-years-out) software projects, called labgrid. In case you have not heard of it yet, feel free to read our short blog post from 2017 or visit labgrid's GitHub page.


Pengutronix at FrOSCon 2018

This year, a team from Pengutronix attended FrOSCon in St. Augustin for the first time. We took the opportunity to shake hands, talk about our latest developments and meet hackers interested in working with embedded Linux.


USB-SD-Mux: Automated SD-Card Juggler

Once the bootloader on your embedded device is up and running the development of kernel and userland in PTXdist-based BSPs is usually based on booting from network. Thus there is no need for the developer to write the boot media with a new image.


PTXdist: Did you know? Today: Just a reboot

While the development on an embedded system I need to reboot it quite often. Doing so I appreciate to keep the required steps as less as possible and be sure the embedded system uses the recently changed data in a consistent manner.


PTXdist: Did you know? Today: Beautify your developer's life

Simplify and beautify your developer's life. An example.