QM meets CI

Marie Mann | | talk

"Hey Marie, how do I schedule my vacation dates again?", since I'm working in the management division of Pengutronix, I know these kind of questions very well. Writing down the processes enables me to answer "RTFM". Processes are organizational tasks that need to be performed over and over again, sometimes based on a fixed interval - sometimes on demand.

When the process documentation at Pengutronix was revised from scratch in 2018, the main question was how to involve the employees in the creation of the processes. For the documentation, a workflow and tools were chosen that were already established among the majority of employees.

This article presents the chosen tools, shows the workflow and explains the structure of a process description as an example.

Why write down processes?

As soon as a company is large enough that not all employees are sitting in the same room and can see which tasks are being worked on at any given time, it becomes necessary to write down precisely these processes. Process documentation makes it possible to record the knowledge about the processes and to establish rules.

Although the rules are set by the management, they are visible to all employees and can therefore be demanded. This enables fair treatment of all employees.

How can employees be involved?

Writing down the processes is not an end in itself, but serves to enable employees to read up on how to apply for vacation, for example.

Often the process users are the content experts, because they follow the process every day and live with it. They know best how a process can be optimized. This fact alone should be reason enough to involve the process users in the design of the processes:

  • the processes can be improved by the users and
  • the processes will be followed much more willingly if they make sense to the users.

Getting started in software development is often compared to a recipe.

A cookbook could also be called process documentation: a process, like cooking, has a trigger (hunger) or entry point and describes a set of instructions for action.

If the process documentation is the source code that runs the business, why not use the tools from software development to write the process documentation?

At Pengutronix, the train of thought was taken seriously and the process descriptions are designed in the form of a process manual using the tools of software development.

For the implementation, there were the following requirements for the processes and their presentation:

  • the processes should be easily accessible,
  • the employees should have an opportunity to participate and be involved in the design of the processes,
  • the processes are (therefore) subject to an approval process by the management,
  • the processes should be regularly checked to ensure that they are up to date.

Tools and workflows

In order to meet the requirements and implement the process manual as a "software project", both workflow and tooling, which are already established in the company, are used. These workflows and tooling are briefly described below. The description will be reduced to the essential features that are necessary for the design of the process documentation.

Linux kernel workflow

One of the most important and widely used workflows at Pengutronix is the one used to develop the Linux kernel.

The Linux kernel is divided into different subsystems, which are roughly comparable to individual departments of a company. Depending on the complexity, there may be further subsystems. Possible changes to the Linux kernel (bug fixes, new drivers, code quality improvements) are first discussed within the subsystem. A maintainer is responsible for each subsystem (this can be compared to a middle management level), who carries out tests and checks whether the quality standards are met. If everything is in order, the changes of all subsystems flow into the next version of the Linux kernel. This is released by Linus Torvalds (the first author and maintainer of Linux) himself.

We have adapted the described workflow for the work on the process manual: changes to the process documentation are discussed with the respective process users and then passed on to the management for approval. Currently, the Pengutronix process manual project is still small enough that "subsystem maintainers" e.g. a middle management layer is not required.

Version control

In order to be able to implement this workflow, the processes are written as structured text files that are managed in a version-controlled manner. The release branch of the version management can only be written for the management, so that a new release of the process manual also ensures the release at the same time.

The advantage of files written in reStructuredText (.rst) or Markdown (.md) plain-text formats as opposed to non-text-based formats such as .docx or .odf is that the change in content can be easily tracked in a version management system along with a timestamp and the respective authors. Furthermore, in addition to just tracking changes, a version management system also allows descriptive text to be written to record why a change was made and what decisions led to the process change.

Web representation

As great as the charm of text files is from the point of view of the process writers, for the process users the user-friendliness is not ideal. Therefore, Pengutronix uses the open source software documentation tool Sphinx to generate a web view of the process manual. In this, the processes can be linked directly as well as forms belonging to a process can be made available for download.

Build server

With the help of a build server, the current release of the process documentation is built and made available as a web view. At the same time, the build server enables the use of automated tests.

At Pengutronix, the build server checks, among other things, whether the date on which the next process review is due has been exceeded. If this occurs, reminder e-mails are sent so that the process can be checked promptly to ensure that it is up to date.

The setup has only been described schematically here, as the focus is not on concrete products but on their use for process documentation. What is more important is that any form of version management is used to track changes and that any kind of build server is used for continuous testing and continuous deployment.

Structure of a process description

In order for the setup to be used to its full extent, it is crucial that a structure for the handbook and the structure of a process is determined in advance. A good example of the structure of processes and how such a manual can be designed is the "Company Handbook" by GitLab Inc.

In order to go into practice, the structure of a process in the process documentation of Pengutronix is described in excerpts, and a test is presented as an example.

A process consists of three sections: the first section is a table in which metadata is recorded. When was the process last updated? When is the next review due? Is there a predecessor process? Which process follows on from the current process? Are there any forms to download?

The second section contains the actual process description. There is no fixed form for this. To make the process easy to read, additional structuring elements such as info boxes or enumerations are used.

The third section is reserved for general information based on the quality criteria of ISO 9001. Among others, the following questions should be answered: Who are the process users and owners? When is the process activated? What requirements are there for the process users and what quality criteria are there for the process to fulfill?

On the one hand, the structure is aimed at the process users so that the information can be found as easily as possible. On the other hand, the metadata table allows the use of test automation.

Test automation

One of the most important uses for test automation is a regular reminder that processes are due for review.

In order for certain elements, such as the next review date of a process, to be reviewed, it is necessary to specify very precisely where the element can be found. The table structure is particularly suitable for this.

The tests here were written in Python and the docutils library was used to parse the file. This code example shows the test for the review date of a process.

def check_Wiedervorlage(self, document):
     assert 'Wiedervorlage' in self._data, \
         annotate(self._table, "metadata table missing 'Wiedervorlage' row")
     value = self._data['Wiedervorlage']
     match = re.match(r"(\w+): ([\d\-]+)", value)
     assert match, \
         annotate(self._table, "metadata 'Wiedervorlage' row has the wrong format")
     groups = match.groups()
         dt = datetime.strptime(groups[1], "%Y-%m-%d")
     except ValueError:
         dt = None
     assert dt is not None, \
         annotate(self._table, "metadata review time ('{}') format is invalid".format(groups[1]))
     assert dt >= datetime.strptime(datetime.now().strftime("%Y-%m-%d"), "%Y-%m-%d"), \
         annotate(self._table, "metadata review time ('{}') is in the past".format(groups[1]))

This test consists of three test steps: first, it checks whether the table column was found at all, then whether the date corresponds to the expected format, and finally whether this date is in the past.


The process documentation has been successfully managed in this form at Pengutronix for three years now and is used by developers, admins as well as the administration (partly in independently managed documentations).

The goal of enabling employees to participate in the design of processes has been achieved. Of course, the process manual, like a software project, is never finished. Therefore, it is also important for the maintenance of the process documentation that there is a responsible person with a corresponding time budget.

Further Readings

Pengutronix at ESE Kongress

The Embedded Software Engineering Kongress is also taking place online this year and we would like to take the opportunity of easy participation to watch the lectures, join in the discussions and get into conversation.

#FlattenTheCurve – Introducing Our Remote Setup

The Corona crisis is a challenge that has hit many people as well as most companies quite unexpectedly. The entire team of Pengutronix wants to thank those that currently ensure our essential supplies, health system and civil infrastructure!

CLT-2022: Voll verteilt!

Marie Mann | | clt, event, linux, talk, tutorial

Unter dem Motto "Voll verteilt" finden die Chemnitzer Linux Tage auch 2022 im virtuellen Raum statt. Wie auch im letzten Jahr, könnt ihr uns in der bunten Pixelwelt des Workadventures treffen und auf einen Schnack über Linux, Open Source, oder neue Entwicklungen vorbei kommen.

Wir haben doch etwas zu verbergen: Schlüssel mit OP-TEE verschlüsseln

Moderne Linux Systeme müssen häufig zwecks Authentifizierung bei einer Cloud- basierten Infrastruktur oder einer On-Premise Maschine eigene kryptografische Schlüssel speichern. Statt der Verwendung eines Trusted Platform Modules (TPM), bieten moderne ARM Prozessoren die TrustZone-Technologie an, auf deren Basis ebenfalls Schlüssel oder andere Geheimnisse gespeichert werden können. Dieses Paper zeigt die Nutzung des Open Portable Trusted Execution Environments (OP- TEE) mit der Standardkonformen PKCS#11 Schnittstellen und i.MX6 Prozessoren von NXP als Beispiel.

Conferences 2021: A Retrospective and Outlook

Marie Mann | | conference, talk

Besides the improvements around embedded Linux software and ongoing development of Linux kernel components, the Pengutronix team took the corona pandemic as an opportunity: Last year, many conferences were accessible via internet, so we joined from the comfort of our home offices and shared our experiences and ideas with the community.