The LXA IOBus line of lab automation devices
I would like to present to you the LXA IOBus, a CAN-based ecosystem consisting of a protocol, a gateway server and new class of Linux Automation GmbH devices, including the Ethernet-Mux and the 4DO-3DI-3AI input/output board.
We at Pengutronix use LXA IOBus devices when we want to automate low-bandwidth interaction with devices under test (DUTs) in our Remotelabs. Two examples for such devices are presented below, the Ethernet-Mux and the 4DO-3DI-3AI.
The Ethernet-Mux allow you to connect the DUT's Ethernet port with one of two upstream Ethernet ports.
The 4DO-3DI-3AI allows the automation of button presses or setting/releasing of a DUTs jumper connections. It also allows reading back digital and analog voltage information from a DUT.
"What's a Remotelab?" You may ask. You may want to have a look at our blog post about our Remotelab infrastructure.
The IOBus protocol
Why build a custom protocol when USB, Ethernet, Modbus and 1-Wire are already available?
USB - Our Remotelabs are equipped to connect up to 40 USB devices per rack. Multiplying that by about eight full-scale Remotelabs currently in use at Pengutronix means that we encounter even the rarest USB-related issues quite regularly. This more often than not means losing contact to to all devices connected to hub until it is manually power-cycled. This quickly became a major time-sink and annoyance to our developers.
Don't get us wrong, we still like USB, whenever its high bandwidth capabilities are required. We also continue to develop new products using USB, some of which even speak USB on all of their ports. But that is a story for another time.
Modbus - Modbus is widely used in industrial automation and there is a selection of reasonably priced products from different manufacturers. We did however not want to commit to it as a protocol for our own hardware developments due to it's limited bandwidth and dated protocol.
Ethernet - Ethernet requires point-to-point connections between devices using rather clunky cabling, making it unwieldy when used for all devices in our Remotelabs. There is also additional software complexity required to implement the different protocol layers from Ethernet to TCP. We are keeping an eye out for possible solutions to these issues, like 10Base-T1S and CoAP, but it will take some time until they can be widely used.
1-Wire - 1-Wire was our protocol of choice before we started developing the LXA IOBus. The proprietary nature of the protocol coupled with the few off-the-shelf tools did however lead to lots of hours spent debugging the setups. The low maximum datarate is also a limiting factor when designing new products.
The lessons learned from our experience with the protocols above led use to choosing a CAN-based protocol for our future lab automation products. More specifically we decided to base the LXA IOBus on CANopen, a device-abstraction layer on top of CAN.
CAN and CANopen have a couple of benefits:
- Rock solid - CAN was designed with safety-critical systems, like cars, in mind. While LXA IOBus devices are not used in safety-critical environments (please don't) this is still a big plus for us. Automatic retransmits at the hardware layer and the automatic disabling of misbehaving nodes means that a single broken device or intermittent failure does not break the whole bus.
- Tooling - CAN is a first-class citizen in the Linux Kernel and there is a large ecosystem around it. You can use Wireshark to debug CAN busses and can even unpack CANopen with it. There is also software to send and receive CAN packets included in most major distributions.
- Multi-drop cabling - CAN, at low datarates, is very forgiving when it comes to cabling. A complete Remotelab can be wired using ribbon- and off-the-shelf D-Sub 9 cables without requiring any hubs or switches. We use a single D-Sub 9 connector to supply data and power to the device.
- Automatic discovery - CANopen allows hot-plugging and the automatic configuration of connected devices.
Please note that our LXA IOBus devices uses vendor-specific CANopen device profiles, so integrating them into your elevator infrastructure or garbage truck electronics may require some extra effort (feel free to contact as if you want to go that route though, we would like to know).
The IOBus Server is the gateway between the IOBus and you. On one side it uses the Linux SocketCAN interface to communicate with devices on the CAN bus, on the other side it provides a HTTP-Server with an interactive web interface and scriptable REST-API.
The IOBus Server is, and always will be, open source and is well integrated into the Labgrid lab automation framework.
The Ethernet-Mux allows the multiplexing of two upstream Ethernet ports to one downstream port (or vice versa). The connections are made using high-speed analog switches, allowing you to use Ethernet at speeds up to 1000Base-T without the added latency that would for example be added by a network-level Ethernet switch.
An example use case is a DUT that should be tested as part of a larger infrastructure, but should also be able to connect to another network for software deployment or remote access.
The LXA IOBus 4DO-3DI-3AI is our first device to break the cycle of LXA (Linux Automation GmbH) devices called Something-Something-Mux, even though it could be used as a GPIO-Mux. But let's not get into that.
The 4DO-3DI-3AI provides four digital outputs, three digital inputs and three analog inputs. The digital inputs and outputs are galvanically isolated and implemented as solid state relays, allowing a lot of flexibility in their use. The analog inputs are not isolated but can be used to measure voltages up to 12V, allowing you to measure most supply rails of an embedded device.
The major use case for the 4DO-3DI-3AI is to programmatically bridge jumper connections like boot mode selectors or reset pins, but it can also be used to simulate button presses by soldering connections to said button.
The inputs can be used as a feedback source in automatic tests to determine if a specific test ran successfully.
A firmware upgrade is due. A newly implemented feature needs to be rolled out, a security issue patched or new hardware support added. The software, while capable, is complex. Pengutronix' strategy to handle this complexity is working on a version- controlled Board Support Package (BSP) with continuous updates and tests on the latest mainline Linux kernel.
Project work with our customers includes the handling of hardware prototypes. Since work is generally done in parallel, on many project for many customers, there is a constant flood of hardware prototypes accumulating on the desks of our developers. These accumulations of loose boards can become a problem. This is especially the case when a number of people work on a prototype. Another common annoyance occurs when a project has not been worked on for a period of time, as this might involve moving the hardware from one desk (or storage location) to another and setting it up again. Right now, in a situation where working from home is more common and relevant than ever, this has become even more of an issue. The distances between desks and storage locations of our developers are now measured in kilometers, rather than meters.
About 70,000 patches go into the Linux kernel every year, and many of them are bug fixes. The same applies to most other open source projects that are part of a modern Linux system. In order to benefit from the work in the community, the sensible strategy is to constantly update to the latest software version and keep the system up to date. Of course, with this amount of changes, new bugs can be added or incompatibilities can arise.
"FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. In 2021, they will gather online." -- FOSDEM
Now that, due to the COVID-19 pandemic, everyone has gotten used to digitalisation and online conferences - it has never been easier to organise a conference and bring together all experts and interested parties for a few hours of intensive exchange of ideas on a certain topic.
Once the bootloader on your embedded device is up and running the development of kernel and userland in PTXdist-based BSPs is usually based on booting from network. Thus there is no need for the developer to write the boot media with a new image.