Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.

How can we support you?

By using Continuous Testing (CT) on real hardware, we are able to detect and fix occurring problems at an early stage. Our development projects essentially consist of two phases:

Initial Product Development

Many projects start with an initial development phase: In this phase we support you in commissioning your hardware, as well as building a Board Support Package (BSP), which is exactly tailored to your hardware. In this phase we develop for example missing drivers, add missing functions to software components or optimize the boot time of your device. In this phase software components are usually constantly updated to the latest versions. At the same time we offer to develop tests for your hardware and your BSP. These tests check the core functions of the system during development. This phase usually ends with the release of your product.

Long Term Maintenance

In addition, we offer you a permanent maintenance of your BSP: We regularly (e.g. several times a year) update your system and ensure with the tests that the functions are still given.

The subsequent maintenance provides you with significant advantages: If a software component should cause an error in your product or a security gap should occur, you will always have an up-to-date system at your disposal. This enables you to have short release cycles. At the same time, the changes from update to update are smaller, so that the actual effort for maintenance is lower.

Please do not hesitate to contact us: In combination with your product knowledge, we will come up with a development and maintenance concept for your operating system components that is tailored to your requirements.

Real World Example

In the following example, a customer relies on Verified Boot for its product. A chain of cryptographic operations ensures that only an operating system signed by the manufacturer can be run on a device. Each component (e.g. the boot loader) checks whether the next software component to be loaded (e.g. the Linux kernel) has a valid signature. Only if this is the case, the respective next component is executed.

Further Information on Verified Boot

Our colleague Marc Kleine-Budde gave the overview talk "Verified Boot: From ROM to Userspace" at ELC Europe 2016. (Youtube, Slides)

The Verified Boot chain, from the configuration of the ROM code of the processor, over the boot loader, the Linux kernel up to the user space, were integrated into the BSP and are still being maintained by Pengutronix.

Test Case

To ensure that the verification of both the FIT image (including Linux kernel and device tree) and the dm verity protected file system work, they are tested. To do this, the FIT image is modified and an attempt is made to boot the system. In a different test, the file being executed as the init process in the dm-verity-protected file system is modified. Reading and executing this file is expected to fail, since the dm-verity hash tree is corrupt. In both scenarios, the boot attempt fails, thus Verified Boot works as expected.

During this test, various functions of the barebox boot loader are tested as a side effect: Images are copied between partitions, file systems are mounted and modified and the triggering of the watchdog is tested.

Error Case

During a maintenance update of the BSP, the boot loader has now been updated and the previously discussed test cases failed. The good news: Verified Boot still worked: The modified system was not executed.

But: The test revealed two bugs in barebox: One bug occurred when copying to POSIX device-files (Fix by Sascha Hauer). The other bug concerns the watchdog (Fix by Ahmad Fatoum). It is expected to reset the system after the Verified Boot chain was corrupted.

Both errors were uncovered by the test suite and the appropriate fixes were integrated into the BSP before the new BSP version was delivered to the customer.

Back to technical showcases

Further Readings

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.


Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.


Linux Automation Test Automation Controller: A one Device labgrid Exporter

Our subsidiary Linux Automation GmbH introduces the LXA TAC (Linux Automation Test Automation Controller): an all-in-one labgrid exporter. The LXA TAC offers the usual interfaces to control one or more embedded devices (DUTs, devices under test) interactively or automatically with labgrid.


Yes we CAN... add new features

Have you ever experienced an otherwise fine product that is missing just the one feature you need for your application?


DSA in Barebox

The v2022.05.0 Release of barebox introduced initial support for the Distributed Switch Architecture (DSA) Framework. DSA is originally a subsystem from the Linux Kernel, which exposes the individual ports of a network switch IC as virtual network interfaces.


Pengutronix at Embedded World 2022

Welcome to our booth at the Embedded World 2022 in Nürnberg!


First Steps using the candleLight

So you went and got yourself one of our fancy rocket-penguin branded CandleLight dongles or, being the die hard hacker you are, went and soldered one up in your toaster oven labeled "not food safe". What's next then? How do you use this thing? Let's answer these question by grabbing a Raspberry Pi and exploring some of the possibilities.


labgrid Tutorials

This week, we started our series of YouTube labgrid tutorials. In the next few weeks we will publish more video tutorials showing you labgrid's features and giving you handy tips and tricks.


The LXA IOBus line of lab automation devices

I would like to present to you the LXA IOBus, a CAN-based ecosystem consisting of a protocol, a gateway server and new class of Linux Automation GmbH devices, including the Ethernet-Mux and the 4DO-3DI-3AI input/output board.


Pengutronix at FOSDEM 2021

"FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. In 2021, they will gather online." -- FOSDEM