Tutorial: Start With RAUC Bundle Encryption Using meta-rauc

GitHub Pull Request

The initial discussion about the feature took place in issue #633.

The implementation done for a Pengutronix customer was proposed and merged in PR #863.

In its current master branch, RAUC now supports encrypted Bundles. This tutorial will introduce you to the basics of using encryption in RAUC and show how to use it in a simplified Yocto setup with the meta-rauc Layer.

Use Case Considerations

RAUC allows encrypting update bundles to prevent third parties from reading the bundle's content and thus from gaining information about potentially sensitive data such as configuration or custom firmware.

In contrast to transport encryption (HTTPS), an encrypted bundle protects its contents independently directly and can thus be used for unprotected transport mediums and channels like HTTP, CAN, USB, etc. but also on potentially insecure storage like cloud servers or USB sticks.

Before using encryption, I would recommended to carefully think about your actual use case, make a threat analysis and have a concept for:

  • key deployment and handling during manufacturing
  • private key storage and protection on hardware
  • key validity, key scope and revocation

Implementation Background

RAUC implements encryption with a new bundle format, called 'crypt'. The 'crypt' format is based on the 'verity' format (that in turn is the successor of the original 'plain' format). Like the 'verity' format, 'crypt' makes use of the Linux kernel's device mapper subsystem for decryption (dm-crypt) and verification (dm-verity) to have transparent, block-based random access to the local or remote bundle payload. This also allows using encryption with RAUC's native HTTP(S) streaming support that requires no intermediate storage space on the device.

The symmetric AES key used for encrypting/decrypting the payload is stored in the bundle's manifest (as the CMS payload of the signature). To protect this key and thus fully encrypt the bundle, the manifest is encrypted using asymmetric CMS encryption.

Using CMS-based asymmetric encryption allows encrypting for one or more different recipients with individual keys, mitigating part of the risk caused by key disclosure.

Because the generation of the base bundle and the encryption to the final recipients are decoupled from each other, the encryption process in RAUC is a two-step one:

  1. Generation of a 'crypt' bundle with symmetrically encrypted payload

    This is typically the result of a BSP build and requires knowledge about the image content and the type of platform it targets, but it requires no information about the individual devices.

  2. Asymmetric encryption of the bundle's manifest and signature for one or more recipients

    This typically happens in a later step and requires the public keys of the individual devices or device groups out in the field.

Private Key Storage

One of the most critical aspects in encryption is of course the protection of the private key. In contrast to verification, where the private signing key is only used on the build host and can be well protected, for encryption another private decryption key must be stored securely on the target device where using interactive password or PIN entry would not be an option.

Depending on the threat model and the level of security required, it might be sufficient to have the key stored as plain PEM file on the target. This can be of interest if the access to the transport medium (USB stick, public server) is much easier than the access to the device's storage.

However, for a security-sensitive application, the private key needs better protection. Depending on your hardware, the private key can be stored in a HSM, TPM, or TEE. RAUC can then access these keys on the target via a PKCS#11 API.

Prepare for Using Encryption Support in meta-rauc

To follow this tutorial, you may use your own bundle recipe, create a new one, or just use the example bundle from the meta-rauc-qemux86 layer of meta-rauc-community.

The example snippets here will mainly use meta-rauc-qemux86. A tutorial on how to set this up for using it with RAUC can be found on our blog.

Note

At the time of writing, encryption support is not part of a RAUC release, yet. This requires using the development version of RAUC.

You can enable building the development version by adding to your local.conf:

RAUC_USE_DEVEL_VERSION = "1"

Encryption support will be available with the upcoming RAUC 1.7 release.

To have the rauc native tool available in your host system for use with oe-run-native, ensure it is added to the native sysroot:

$ bitbake rauc-native -c addto_recipe_sysroot

Prepare Target System for Encryption

In this section, you will adapt your BSP to build an image for your target system that is prepared for encryption support.

We need to add the section below to our existing RAUC system configuration file system.conf. In meta-rauc-qemux86, this means editing recipes-core/rauc/files/qemux86-64/system.conf.

[encryption]
key=crypt-key.pem
cert=crypt-cert.pem

This tells RAUC where to find the private key and cert for decrypting bundles. Providing the cert is not mandatory, but, especially for setups with a large number of recipients, it speeds up finding the proper recipient significantly.

Note

For the sake of simplicity, we use plain files in our example. When using PKCS#11 API as recommended, this would look like

[encryption]
key=pkcs11:token=rauc;object=crypt-key
cert=pkcs11:token=rauc;object=crypt-cert

The actual key and cert file will not be placed in the BSP as we use per-device keys and thus add them during device bring-up in the 'factory'.

As previously mentioned, RAUC uses the kernel device mappers dm-crypt and dm-verity for crypt bundles. Support for these must be enabled in the kernel. Thus make sure your kernel is compiled with:

CONFIG_MD=y
CONFIG_BLK_DEV_DM=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_DM_VERITY=y
CONFIG_SQUASHFS=y
CONFIG_CRYPTO_SHA256=y
CONFIG_DM_CRYPT=y

Note

For the tutorial, also make sure you also have an SSH server installed and running on your target system. We will use this for copying files later.

Now we can build the target system's rootfs. For meta-rauc-qemux86 this would mean running:

$ bitbake core-image-minimal

We can then start our system. Use a separate shell for this so we can continue working with our current BSP environment.

For the QEMU machine, this would mean running:

$ source poky/oe-init-build-env build
$ runqemu nographic slirp ovmf wic core-image-minimal

Building A 'crypt' Bundle

In this section, we will now perform step 1 of the two-step process of creating an encrypted bundle.

Building a 'crypt' bundle is as simple as setting your bundle type to 'crypt' in the manifest used for generation. With meta-rauc, you can set this in you bundle recipe file (recipes-core/bundles/qemu-demo-bundle.bb for meta-rauc-qemux86) with:

RAUC_BUNDLE_FORMAT = "crypt"

Note

Since we will use decryption keys that are not part of the BSP build, we need to ensure that these keys are transfered to the updated system. (In our demo this is not strictly required as we do not intend to update from the just-installed system again).

A simple solution to this is to use a post-install hook in our bundle that copies the key and certificate files to the target slot after having written it.

An example hook file would look like:

#!/bin/sh

set -ex

case "$1" in
        slot-post-install)
                # only rootfs class is valid to be handled
                test "$RAUC_SLOT_CLASS" = "rootfs" || exit 1

                cp /etc/rauc/crypt-key.pem $RAUC_SLOT_MOUNT_POINT/etc/rauc/
                cp /etc/rauc/crypt-cert.pem $RAUC_SLOT_MOUNT_POINT/etc/rauc/
                ;;
        *)
                exit 1
                ;;
esac

In your bundle recipe, you can add this by placing it in your bundle recipe's file directory and setting:

SRC_URI += "file://hook.sh"
RAUC_BUNDLE_HOOKS[file] = "hook.sh"
RAUC_SLOT_rootfs[hooks] = "post-install"

Note that the settings may differ when not using meta-rauc-qemux86.

And then just build your bundle:

$ bitbake qemu-demo-bundle

This will generate what we call an 'unencrypted crypt bundle', reflecting that the payload is encrypted, but the CMS holding the manifest and thus the dm-crypt symmetric AES key isn't.

Important

For a real scenario, make sure that this 'intermediate' bundle does not leave your secure environment so that the dm-crypt AES key is not leaked. Anyone knowing the dm-crypt AES key will have full access to the bundle payload!

You can inspect the resulting bundle with rauc info which will give you the hint [unencrypted CMS].

$ oe-run-native rauc-native rauc info --keyring=example-ca/ca.cert.pem tmp/deploy/images/qemux86-64/qemu-demo-bundle-qemux86-64.raucb
Compatible:  'qemu86-64 demo platform'
Version:     '1.0'
Description: 'qemu-demo-bundle version 1.0-r0'
Build:       '20220328213514'
Hooks:       ''
Bundle Format:       crypt [unencrypted CMS]
  Crypt Key:         '<hidden>'
  Verity Salt:       'e6800d82b6d37df11bae194e28abfbea53bd40e2ccd7ac1d85435a2ea47ab552'
  Verity Hash:       '83ab98d255488e96c59b8e5fe122c70b6206b35c4b885f8ca3b8af1a95da7302'
  Verity Size:       512000
[...]

This bundle cannot be deployed and install, yet. We have to fully encrypt it first.

This is the moment when the built bundle normally leaves the development environment and is transferred to your deployment preparation infrastructure. However, the bundle's payload/content is already signed and will not be changed anymore beyond this point!

Create and Deploy Device Keys

In this section, we simulate the factory process of creating and deploying asymmetric per-device keys/certificates that will be used for encryption and decryption.

Tip

The keys and certificates used for encryption do not need to have any relation to the keys and certificates used for the signing process.

To encrypt the bundle for a number of recipients, we typically need to have access to our device management system, which stores the devices' public recipient certificates. All steps described here are normally done before bringing your devices in the field.

In general, it is up to you and your use case how finely-grained you need to encrypt your bundle. Here we assume that each device has its own individual key. This allows revoking a device's key if it should be compromised. Future bundles will then be encrypted for all but the compromised key. See more about use cases and workflows in the RAUC documentation here

While RAUC potentially allows encrypting a bundle to a large number of recipients, this minimal example will use only a few recipients for the sake of simplicity.

Let's create some ECC key pairs with certificates first:

$ mkdir -p keys/private
$ for i in $(seq 0 4); do
> openssl ecparam -name prime256v1 -genkey -noout -out keys/private/private-key-$i.pem
> openssl ec -in keys/private/private-key-$i.pem -pubout -out keys/public-key-$i.pem
> openssl req -new -x509 -key keys/private/private-key-$i.pem -out keys/recipient-cert-$i.pem -days 365250 -subj "/O=RAUC/CN=RAUC Test $i"
> done

This will generate you 5 individual encryption key pairs in the keys/ directory where the sensitive private keys are stored in the keys/private/ directory.

We use a single key pair as decryption keys for our demo target, the others could be used for other targets. Simply copy these key pairs to the devices' root filesystems (note that the final file names must match those we configured in system.conf):

$ scp -P 2222 keys/private/private-key-0.pem root@localhost:/etc/rauc/crypt-key.pem
$ scp -P 2222 keys/recipient-cert-0.pem root@localhost:/etc/rauc/crypt-cert.pem

Your target's rootfs /etc/rauc directory should now look as follows:

root@qemux86-64:~# ls -1 /etc/rauc
ca.cert.pem
crypt-cert.pem
crypt-key.pem
system.conf

Your target is now fully capable of decrypting and installing RAUC bundles.

Encrypt Bundle For Our Recipients

Now that we have our target ready, in this section we will perform step 2 of the two-step bundle encryption: Creating a fully CMS-encrypted 'crypt' bundle for our target and other potential recipients.

To allow encrypting for a huge number of recipients, RAUC supports loading the recipient certificates as a concatenated list from a single file.

To create such, we simply run

$ cat keys/recipient-cert-0.pem keys/recipient-cert-2.pem keys/recipient-cert-4.pem > keys/recipients.pem

Note

We intentionally omit recipient-cert-1.pem and recipient-cert-3.pem here for later purposes.

Now, we can encrypt our bundle using the new rauc encrypt command:

$ oe-run-native rauc-native rauc encrypt \
  --keyring=example-ca/ca.cert.pem \
  --to keys/recipients.pem tmp/deploy/images/qemux86-64/qemu-demo-bundle-qemux86-64.raucb \
  encrypted.raucb
Encrypted bundle written to encrypted.raucb

Note

We could also specify each recipient cert separately by using --to multiple times, but this quickly becomes inconvenient, especially with larger numbers of recipients.

The resulting encrypted.raucb is now a fully encrypted crypt bundle.

A simple rauc info call (as already done above) should not show any information anymore.

$ oe-run-native rauc-native rauc info --keyring=example-ca/ca.cert.pem encrypted.raucb
Encrypted bundle detected, but no decryption key given

Showing the bundle information is now only possible with one of the decryption keys given! We provide it and add the --dump-recipients argument to make the list of recipients of this bundle visible. Note that the Bundle format is now printed as [encrypted CMS].

$ oe-run-native rauc-native rauc info --keyring=example-ca/ca.cert.pem --key=keys/private/private-key-0.pem --dump-recipients encrypted.raucb
Compatible:  'qemu86-64 demo platform'
Version:     '1.0'
Description: 'qemu-demo-bundle version 1.0-r0'
Build:       '20220328213514'
Hooks:       ''
Bundle Format:       crypt [encrypted CMS]
  Crypt Key:         '<hidden>'
  Verity Salt:       'e6800d82b6d37df11bae194e28abfbea53bd40e2ccd7ac1d85435a2ea47ab552'
  Verity Hash:       '83ab98d255488e96c59b8e5fe122c70b6206b35c4b885f8ca3b8af1a95da7302'
  Verity Size:       512000

[...]

3 Recipients:
  0   Issuer:    O = RAUC, CN = RAUC Test 0
      Serial:    0x3078C175A64B65BAADC9751FF4B52624D98CDE01
      Algorithm: dhSinglePass-stdDH-sha1kdf-scheme
  1   Issuer:    O = RAUC, CN = RAUC Test 2
      Serial:    0x61999A7BBC9B9FE3F05CB100B6E4D1899F699782
      Algorithm: dhSinglePass-stdDH-sha1kdf-scheme
  2   Issuer:    O = RAUC, CN = RAUC Test 4
      Serial:    0x620632AECA563A343A35BC53205021732C58AE19
      Algorithm: dhSinglePass-stdDH-sha1kdf-scheme

Install Encrypted Bundle on the Target

Now, we have everything prepared to actually install the bundle on the target. This is exactly what we will do in this final section.

First, copy the bundle to the target.

$ scp -P 2222 encrypted.raucb root@localhost:/data/

Note

We could also install the bundle directly over HTTPS by making use of RAUC's streaming capabilities, but since this requires us to set up an HTTP server first, we skip this for the sake of simplicity.

Then, on the target, simply run rauc install to install your encrypted bundle.

root@qemux86-64:~# rauc install /data/encrypted.raucb
installing
  0% Installing
  0% Determining slot states
 20% Determining slot states done.
 20% Checking bundle
 20% Verifying signature
 40% Verifying signature done.
 40% Checking bundle done.
 40% Checking manifest contents
 60% Checking manifest contents done.
 60% Determining target install group
 80% Determining target install group done.
 80% Updating slots
 80% Checking slot efi.0
 85% Checking slot efi.0 done.
 85% Copying image to efi.0
 90% Copying image to efi.0 done.
 90% Checking slot rootfs.1
 95% Checking slot rootfs.1 done.
 95% Copying image to rootfs.1
100% Copying image to rootfs.1 done.
100% Updating slots done.
100% Installing done.
idle
Installing `/data/encrypted.raucb` succeeded

Finally, to see that this is not all just for show but actually works, let's exchange the decryption key with one we did not sign the bundle for:

$ scp -P 2222 keys/private/private-key-3.pem root@localhost:/etc/rauc/crypt-key.pem
$ scp -P 2222 keys/recipient-cert-3.pem root@localhost:/etc/rauc/crypt-cert.pem
root@qemux86-64:~# rauc install /data/encrypted.raucb
installing
  0% Installing
  0% Determining slot states
 20% Determining slot states done.
 20% Checking bundle
 40% Checking bundle failed.
100% Installing failed.
LastError: Failed to decrypt bundle: Failed to decrypt CMS EnvelopedData: error:1C800066:Provider routines::cipher operation failed
Installing `/data/encrypted.raucb` failed

As expected, the bundle decryption fails and thus the installation is aborted with an error.

You can now play around with encryption support on your own, integrate it into your setup (if you haven't done already while following the tutorial), discuss on our community channel IRC/Matrix or on GitHub, or contact Pengutronix for professional support.


Weiterführende Links

Tutorial: Evaluating RAUC on QEMU - A Quick Setup With Yocto

RAUC is an update framework for safely deploying verified updates on your embedded Linux devices. It ensures atomicity of the update process to protect from sudden power outages, hardware failures, etc. So, why would one like to run RAUC on an emulated platform?


RAUC v1.11 Released

Ho Ho ho! As the year's progress bar approaches 99%, another update is already completed: RAUC v1.11 is here!


RAUC v1.10 Released

Just in time for the EOSS 2023 in Prague, we have released v1.10 of RAUC. Just-in-time means the release was actually finalized by Jan Lübbe in the train to Prague (like I finally wrote the majority of this blog post on the train back).


RAUC v1.9 Released

"Getting things off the ground" could be the motto for the v1.9 release of RAUC. The support for custom metadata in the manifest got a step further, a new, more flexible, D-Bus API for bundle inspection paved the way for obtaining more detailed information, and a new manifest hash marks the first of several planned changes for configurable event logging. However, one of the most invasive changes happened under the hood: We have started the transition from autotools to meson as a build system.


CLT-2022: Voll verteilt!

Unter dem Motto "Voll verteilt" finden die Chemnitzer Linux Tage auch 2022 im virtuellen Raum statt. Wie auch im letzten Jahr, könnt ihr uns in der bunten Pixelwelt des Workadventures treffen und auf einen Schnack über Linux, Open Source, oder neue Entwicklungen vorbei kommen.


labgrid Tutorials

This week, we started our series of YouTube labgrid tutorials. In the next few weeks we will publish more video tutorials showing you labgrid's features and giving you handy tips and tricks.


Yocto Project Virtual Summit 2021

On Tuesday, 25th and Wednesday, 26th, the 3rd edition of the Yocto Project Virtual Summit took place on the internet. With a fair ticket price of 40$ Pengutronix developers Jan Lübbe and Enrico Jörns got a 2-day long wild ride through the latest features, workflows and experiences with the Yocto Project.

Pengutronix wird Yocto Project Participant

Nicht zuletzt da das Yocto Projekt aus der Embedded-Welt schon seit längerem nicht mehr weg zu denken ist, setzt auch Pengutronix bereits seit einigen Jahren neben PTXdist in diversen Projekten auch OpenEmbedded erfolgreich als Distributions-Buildtool ein.