Showcase: Remote Working

Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.

In this article we would like to present to you our Remotelab infrastruture. It was initially developed for Continuous Testing, but has now shown to be one of our main enablers for remote working, allowing us to stay productive with less than a handful of colleagues on-premise.

Our Remotelabs consist of racks housing embedded prototype boards by our customers. Additional infrastructure in our Remotelabs enables our developers to remote control these boards (see below). Once the hardware is located in our infrastructure, it can also be tested, automatically and continuously. This allows us to be alerted about any breaking changes affecting our customers.

A few words about continuous testing

In fact, the ability to test permanently and automatically is one of the main drivers for our ongoing development of labgrid (the software framework to access the hardware) and the professionalization of our Remotelab infrastructure.

We would like to show you the advantages of continuous testing in our Showcase: Continuous Testing.

How can we supoort you?

As a service provider in the software development environment, we are not in the position of selling you complete and ready-made Remotelab infrastructure, but we would be happy to assist you in such projects: Whether it is about integrating support for your test hardware in labgrid, working out test strategies, developing tests, or integrating labgrid into your processes – do not hesitate contacting us.

Even without investing in your own Remotelab, you can still benefit from our Remotelab infrastructure, as our customers hardware is only a few keystrokes away for our development team at any point in the product cycle.

In the last years we have constructed several Remotelabs and want to share our design process with you.

What components does a Remotelab consist of?

19"-Rack

The supporting component of our Remotelab: We use a 19" TE network cabinet from Rittal. The transport rollers are a particularly useful feature. To protect the hardware, each shelf is covered with an ESD shelf lining. In addition, cable guide rails enable an (at least initially) tidy Remotelab. Two suitable sockets for rack mounting provide the necessary power for the embedded boards.

Network switch

Our favorite in this category is the CBS350-24P-4G from Cicso, as it is PoE capable, silent and managed. By using PoE we want to minimize the need for individual power supplies in future in-house hardware developments. The use of managed switches allows us to respond to any special requirements the customer hardware may have for the network setup.

Quiet operation of the switches is important to us, as some of the Remotelabs are located in the developer offices.

Small Server

A small server is needed to manage each of the devices in the Remotelab. The hardware requirements for this labgrid node are quite low, aspects we pay attention to are:

  • Quiet operation -- Our Remotelab servers use PC Engines APU.2E4, allowing fanless operation.
  • Stable USB -- Large USB installations, such as our Remotelabs are, do, in our experience, carry a lot of potential for frustration. The APU.2E4's internal USB controllers and the EXSYS EX-49010-2 controllers installed in addition have proven to be comparatively reliable in operation.
  • CAN bus integration -- For communication with our own custom hardware and for testing the embedded devices, we use CAN-FD capable miniPCIe cards from Peak System, providing four independent CAN buses.

Switchable socket

Have you tried turning it off and on again? -- The easiest way to restart the hardware is to toggle the supply voltage, for example via network-connected power strips, which can be controlled using labgrid. As a low-cost alternative for smaller setups, we are also gaining initial experience with individual sockets controllable via WiFi.

For use in the Remotelab, where many devices are to be controlled, we use GUDE 8080 socket strips, for individual devices we are experimenting with individual sockets from the manufacturer SONOFF, which are flashed with the free Tasmota firmware.

USB-Hubs

USB is almost inevitable in embedded development, we use it to communicate with the hardware and to upload the software. When looking for reliable USB hubs, it is important to try products from different manufacturers. We have had good experiences with the 10-port UA0096 hubs from LogiLink.

An interesting use case that is unfortunately not covered by these hubs is switching the supply voltage of individual ports on and off using uhubctl.

WiFi router

In order to be able to test Bluetooth and WiFi functionalities of the embedded hardware, each remote lab has its own WiFi router, which has been flashed with the free OpenWRT firmware. This router is not connected to the Internet or any other network and is only used to test the connection setup in automatic tests.

Serial console

For the connection of the serial consoles of the embedded devices we use simple USB-serial converters, on which preferably a CP210x chip from Silicon Labs is used. This chip allows us to store a unique ID in each USB-serial converter easily. We put a note containing this ID on any USB-serial converter, helping us managing a few dozen converters per lab.

Remotely setable jumper pins

In order to be able to set and release reset or boot option jumpers as flexibly as possible, potential-free switches are helpful; here we rely on our own developments connected via OneWire or CAN. Alternatively, USB-controlled relays or professional Modbus devices are also available and usable in labgrid.

Access from remote: Our labgrid Integration

To be able to take full advantage of the Remotelab, the hardware alone is not enough. For the control of the embedded boards from the developer PC and as a basis for tests and development on real hardware, Pengutronix uses the open source tool labgrid as an abstraction layer.

Labgrid provides for automated tests as well as for interactive work an intuitive and at the same time versatile interface for working with Embdded Linux devices.

More information about labgrid

Our colleague Chris Fiege gave the talk "About the joy and tears of testing embedded devices" at FOSDEM 2021 on the state of three years of experience with labgrid testing. (Youtube, Slides)

There are several ways to access embedded boards via labgrid, e.g.:

  • Serial console -- Almost always available, even early in the boot process.
  • Access via network and ssh -- More convenient especially later in the development process.
  • Camera stream of the display -- Relevant during driver development and development of graphical applications.

How does the software get onto the embedded board?

This depends on the corresponding hardware and what is to be tested. For example, the bootloader can load the data via the network, a programming adapter can be connected or the software gets loaded via USB. If possible, however, we use the USB-SD-Mux, that allows switching a micro SD card between the Remotelab server and the prototype hardware.

Back to technical showcases

Further Readings

Showcase: Embedded off-the-shelf

A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.


Showcase: Continuous Testing

About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.


Pengutronix at FOSDEM 2021

"FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. In 2021, they will gather online." -- FOSDEM


Pengutronix at Live Embedded Event

conference, event, testing

Now that, due to the COVID-19 pandemic, everyone has gotten used to digitalisation and online conferences - it has never been easier to organise a conference and bring together all experts and interested parties for a few hours of intensive exchange of ideas on a certain topic.


#FlattenTheCurve – Introducing Our Remote Setup

The Corona crisis is a challenge that has hit many people as well as most companies quite unexpectedly. The entire team of Pengutronix wants to thank those that currently ensure our essential supplies, health system and civil infrastructure!


A Logo for labgrid

Enrico Jörns | | labgrid

It took us a while to find a good logo for one of our latest (but already quite-a-few-years-out) software projects, called labgrid. In case you have not heard of it yet, feel free to read our short blog post from 2017 or visit labgrid's GitHub page.


Pengutronix at FrOSCon 2018

This year, a team from Pengutronix attended FrOSCon in St. Augustin for the first time. We took the opportunity to shake hands, talk about our latest developments and meet hackers interested in working with embedded Linux.


USB-SD-Mux: Automated SD-Card Juggler

Once the bootloader on your embedded device is up and running the development of kernel and userland in PTXdist-based BSPs is usually based on booting from network. Thus there is no need for the developer to write the boot media with a new image.


PTXdist: Did you know? Today: Just a reboot

While the development on an embedded system I need to reboot it quite often. Doing so I appreciate to keep the required steps as less as possible and be sure the embedded system uses the recently changed data in a consistent manner.


PTXdist: Did you know? Today: Beautify your developer's life

Simplify and beautify your developer's life. An example.