Showcase: Preempt RT and Time Sensitive Networking

J. R. R. Tolkien already knew...

"Real Time is never late, nor is it early, it arrives precisely when it means to."

Nowadays, even small and cheap microcontrollers offer enough calculation power to perform time critical tasks within an industrial environment. However, as soon as actors and sensors are spread over an entire facility and are to be connected over Ethernet, the actual moment when a data packet will get processed becomes very hard to predict. At this point, Linux running a Preempt RT Kernel altogether with a network featuring Time Sensitive Networking (TSN) capabilities can help.

When sending data it is likely that delays not only occur inside the processing system, but also on the transmission path. As such, critical data from multiple sources might arrive at the recipient at once. The more modern embedded and connected devices allow for fast calculations and data handling, the more the question arises not only if your system performs "fast enough", but whether your results also arrive "in time".

The usage of complex network stacks based on standard Ethernet may be the motivation to swap the microcontroller with a yet more capable soulution based on a microprocessor running a proper Linux as operating system. Here, Real Time features are implemented by the Preempt RT mechanism of the Linux Kernel, and time critical data transfer also gets covered by more and more TSN features being added over the time.

TSN Demonstrator

Being a very heterogenous topic, TSN for itself consists of a variety of technologies: On one hand, not all standardized processes are already available under Linux, while on the other hand not every implementation even requires the full set of available features.

In order to gather further insights into the technical challenges that occur during the development process we at Pengutronix have designed a TSN Demonstrator.

The "Floating Sphere" is a popular experiment where the basic concept of a PID control loop is demonstrated. In that setup a magnet is attached to a lightweight object (e.g. a paper ring). This shape is then made levitating in the field of an electromagnet: A sensor detects the position of the ball and reports it to a controller, which in return regulates the current through the electromagnet accordingly to a constant distance from the sensor - when done right, the sphere keeps floating in an equilibrium as long as the setup is powered.

To simulate a system that is connected via TSN Ethernet the sensor, actor and controller were realized on single-board computers that are based on the STM32MP1 microprocessor running the latest Linux Kernel (5.11) and Preempt RT. The boards are connected over Ethernet and Precision Time Protocol is used to synchronize the clocks of the participants.

Goals

The Demonstrator setup inherits various latencies when the process of regulating the paper ring's position takes place:

  • The information gathered by the sensor has to be processed by the first system.
  • The packet has then to find its way through the network and to the board with the controller software.
  • The controller must calculate the electromagnet's next state whilst sharing the board resources with other programs and services.
  • The resulting values again are to be sent over the network to finally reach the actor's system to get applied at the right moment.

With the system as a whole now being to slow to maintain a stable floating position the individual components have now to be coordinated in real time.

Subsequently, we realized our PID-loop as follows:

  • By using PTP (IEEE 802.1AS), the internal clocks of our boards were synchronized within in the margin of less than a microsecond.
  • Critical latencies within the software environment were tracked down and solved if necessary.
  • The positioning sensor is being read synchronous to the PTP timer and a respective time stamp gets applied; the daten then needs about 600 µs from sensor to actor.
  • The PID controller runs at a frequency of 1.25 kHz / 800 µs.
  • Lastly, the actor's software has to take care that the new sVtate of the electromagnet is applied at the right time.

The result can be seen in the accompanying pictures.

Next steps

The Demonstrator setup allows for first insights into the different technologies around TSN under Linux. During the development a range of current SoCs was evaluated regarding their TSN capabilities: Not all features are available at every hardware, so the SoC has to be chosen specific to the task at hand. Additional room for improvments also exists on the software-side - many of the used technologies are still young and quickly evolve from Kernel to Kernel. It is therefore recommended to always use a current version of Linux.

How can Pengutronix help with TSN projects?

More about TSN and PTP

Our friends at OSADL are constantly conducting measuring tasks regarding the performance of Real-Time and TSN systems. If you are interested, you may check out their article "Real-time Ethernet (TSN) synchronization analysis". (osadl.org)

The Pengutronix Kernel-Team can advise industrial customers on the development of TSN systems as well as directly support them with development services. An important task is the further development and adaption of the Linux Kernel to the individual requirements of the customer's project, for example through new drivers for switch chips and TSN features in network hardware.

Back to technical showcases

Further Readings

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.


Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.


Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.


labgrid Tutorials

This week, we started our series of YouTube labgrid tutorials. In the next few weeks we will publish more video tutorials showing you labgrid's features and giving you handy tips and tricks.


Wie man (k)ein Betriebssystem für Produkte baut

Distributionen wie Raspbian lassen die passgenaue Zusammenstellung eines Betriebssystems kinderleicht aussehen. Image herunterladen, Pakete installieren, noch ein paar Änderungen - fertig. Alles wie auf dem Laptop oder Server. Warum ein Betriebssystem aus einer klassischen Distribution im Produkt-Kontext zur Katastrophe führen kann, beleuchtet der Vortrag "Raspbian vs. Build-Systeme: Das richtige Werkzeug für solide Produkte".


CLT-2021: Mach es einfach anders!

Marie Mann | | clt, linux, opensource

"Mach es einfach anders!" - unter diesem Motto finden die CLT dieses Jahr im virtuellen Raum statt. Wie auch in den letzten Jahren ist Pengutronix als Sponsor dabei. Anders ist, dass wir dieses Jahr unser eigenes kleines Programm mit spannenden Kurzvorträgen und täglich zwei Quiz-Runden mit tollen Hauptgewinnen mitbringen.


Showcase: Fail-Safe (OTA) Field Updating

Enrico Jörns | | didyouknow, rauc

Being able to robustly and securely update embedded systems and IoT devices in the field is a key requirement of every product today. The update framework RAUC is the basis for a modern and future-proof solution. In this showcase we present the basic principles of a fail-safe update system and how Pengutronix can support you with implement this for your platform.


Showcase: Graphics on i.MX8MP

Enabling the graphics output pipeline on the i.MX8M Plus (i.MX8MP for short), is the most recent example on how open-source and upstream driver support for GPU and display engines can reduce effort and risk in a new project.


eMMC Hardware Partitioning

When designing an embedded system, one must consider both the application and the underlying hardware in combination, if the intended long-term stability is to be achieved. While we discussed the necessity of software updates in previous posts, in this article I describe a way to use a memory subsystem corresponding to its physics to achieve the best retention and lifetime of the whole system.