Tutorial: Start With RAUC Bundle Encryption Using meta-rauc
In its current master branch, RAUC now supports encrypted Bundles. This tutorial will introduce you to the basics of using encryption in RAUC and show how to use it in a simplified Yocto setup with the meta-rauc Layer.
Use Case Considerations
RAUC allows encrypting update bundles to prevent third parties from reading the bundle's content and thus from gaining information about potentially sensitive data such as configuration or custom firmware.
In contrast to transport encryption (HTTPS), an encrypted bundle protects its contents independently directly and can thus be used for unprotected transport mediums and channels like HTTP, CAN, USB, etc. but also on potentially insecure storage like cloud servers or USB sticks.
Before using encryption, I would recommended to carefully think about your actual use case, make a threat analysis and have a concept for:
- key deployment and handling during manufacturing
- private key storage and protection on hardware
- key validity, key scope and revocation
RAUC implements encryption with a new bundle format, called 'crypt'. The 'crypt' format is based on the 'verity' format (that in turn is the successor of the original 'plain' format). Like the 'verity' format, 'crypt' makes use of the Linux kernel's device mapper subsystem for decryption (dm-crypt) and verification (dm-verity) to have transparent, block-based random access to the local or remote bundle payload. This also allows using encryption with RAUC's native HTTP(S) streaming support that requires no intermediate storage space on the device.
The symmetric AES key used for encrypting/decrypting the payload is stored in the bundle's manifest (as the CMS payload of the signature). To protect this key and thus fully encrypt the bundle, the manifest is encrypted using asymmetric CMS encryption.
Using CMS-based asymmetric encryption allows encrypting for one or more different recipients with individual keys, mitigating part of the risk caused by key disclosure.
Because the generation of the base bundle and the encryption to the final recipients are decoupled from each other, the encryption process in RAUC is a two-step one:
Generation of a 'crypt' bundle with symmetrically encrypted payload
This is typically the result of a BSP build and requires knowledge about the image content and the type of platform it targets, but it requires no information about the individual devices.
Asymmetric encryption of the bundle's manifest and signature for one or more recipients
This typically happens in a later step and requires the public keys of the individual devices or device groups out in the field.
Private Key Storage
One of the most critical aspects in encryption is of course the protection of the private key. In contrast to verification, where the private signing key is only used on the build host and can be well protected, for encryption another private decryption key must be stored securely on the target device where using interactive password or PIN entry would not be an option.
Depending on the threat model and the level of security required, it might be sufficient to have the key stored as plain PEM file on the target. This can be of interest if the access to the transport medium (USB stick, public server) is much easier than the access to the device's storage.
However, for a security-sensitive application, the private key needs better protection. Depending on your hardware, the private key can be stored in a HSM, TPM, or TEE. RAUC can then access these keys on the target via a PKCS#11 API.
Prepare for Using Encryption Support in meta-rauc
To follow this tutorial, you may use your own bundle recipe, create a new one, or just use the example bundle from the meta-rauc-qemux86 layer of meta-rauc-community.
The example snippets here will mainly use meta-rauc-qemux86. A tutorial on how to set this up for using it with RAUC can be found on our blog.
At the time of writing, encryption support is not part of a RAUC release, yet. This requires using the development version of RAUC.
You can enable building the development version by adding to your local.conf:
RAUC_USE_DEVEL_VERSION = "1"
Encryption support will be available with the upcoming RAUC 1.7 release.
To have the rauc native tool available in your host system for use with oe-run-native, ensure it is added to the native sysroot:
$ bitbake rauc-native -c addto_recipe_sysroot
Prepare Target System for Encryption
In this section, you will adapt your BSP to build an image for your target system that is prepared for encryption support.
We need to add the section below to our existing RAUC system configuration file system.conf. In meta-rauc-qemux86, this means editing recipes-core/rauc/files/qemux86-64/system.conf.
[encryption] key=crypt-key.pem cert=crypt-cert.pem
This tells RAUC where to find the private key and cert for decrypting bundles. Providing the cert is not mandatory, but, especially for setups with a large number of recipients, it speeds up finding the proper recipient significantly.
For the sake of simplicity, we use plain files in our example. When using PKCS#11 API as recommended, this would look like
[encryption] key=pkcs11:token=rauc;object=crypt-key cert=pkcs11:token=rauc;object=crypt-cert
The actual key and cert file will not be placed in the BSP as we use per-device keys and thus add them during device bring-up in the 'factory'.
As priviously mentioned, RAUC uses the kernel device mappers dm-crypt and dm-verity for crypt bundles. Support for these must be enabled in the kernel. Thus make sure your kernel is compiled with:
CONFIG_MD=y CONFIG_BLK_DEV_DM=y CONFIG_BLK_DEV_LOOP=y CONFIG_DM_VERITY=y CONFIG_SQUASHFS=y CONFIG_CRYPTO_SHA256=y CONFIG_DM_CRYPT=y
For the tutorial, also make sure you also have an SSH server installed and running on your target system. We will use this for copying files later.
Now we can build the target system's rootfs. For meta-rauc-qemux86 this would mean running:
$ bitbake core-image-minimal
We can then start our system. Use a separate shell for this so we can continue working with our current BSP environment.
For the QEMU machine, this would mean running:
$ source poky/oe-init-build-env build $ runqemu nographic slirp ovmf wic core-image-minimal
Building A 'crypt' Bundle
In this section, we will now perform step 1 of the two-step process of creating an encrypted bundle.
Building a 'crypt' bundle is as simple as setting your bundle type to 'crypt' in the manifest used for generation. With meta-rauc, you can set this in you bundle recipe file (recipes-core/bundles/qemu-demo-bundle.bb for meta-rauc-qemux86) with:
RAUC_BUNDLE_FORMAT = "crypt"
Since we will use decryption keys that are not part of the BSP build, we need to ensure that these keys are transfered to the updated system. (In our demo this is not strictly required as we do not intend to update from the just-installed system again).
A simple solution to this is to use a post-install hook in our bundle that copies the key and certificate files to the target slot after having written it.
An example hook file would look like:
#!/bin/sh set -ex case "$1" in slot-post-install) # only rootfs class is valid to be handled test "$RAUC_SLOT_CLASS" = "rootfs" || exit 1 cp /etc/rauc/crypt-key.pem $RAUC_SLOT_MOUNT_POINT/etc/rauc/ cp /etc/rauc/crypt-cert.pem $RAUC_SLOT_MOUNT_POINT/etc/rauc/ ;; *) exit 1 ;; esac
In your bundle recipe, you can add this by placing it in your bundle recipe's file directory and setting:
SRC_URI += "file://hook.sh" RAUC_BUNDLE_HOOKS[file] = "hook.sh" RAUC_SLOT_rootfs[hooks] = "post-install"
Note that the settings may differ when not using meta-rauc-qemux86.
And then just build your bundle:
$ bitbake qemu-demo-bundle
This will generate what we call an 'unencrypted crypt bundle', reflecting that the payload is encrypted, but the CMS holding the manifest and thus the dm-crypt symmetric AES key isn't.
For a real scenario, make sure that this 'intermediate' bundle does not leave your secure environment so that the dm-crypt AES key is not leaked. Anyone knowing the dm-crypt AES key will have full access to the bundle payload!
You can inspect the resulting bundle with rauc info which will give you the hint [unencrypted CMS].
$ oe-run-native rauc-native rauc info --keyring=example-ca/ca.cert.pem tmp/deploy/images/qemux86-64/qemu-demo-bundle-qemux86-64.raucb Compatible: 'qemu86-64 demo platform' Version: '1.0' Description: 'qemu-demo-bundle version 1.0-r0' Build: '20220328213514' Hooks: '' Bundle Format: crypt [unencrypted CMS] Crypt Key: '<hidden>' Verity Salt: 'e6800d82b6d37df11bae194e28abfbea53bd40e2ccd7ac1d85435a2ea47ab552' Verity Hash: '83ab98d255488e96c59b8e5fe122c70b6206b35c4b885f8ca3b8af1a95da7302' Verity Size: 512000 [...]
This bundle cannot be deployed and install, yet. We have to fully encrypt it first.
This is the moment when the built bundle normally leaves the development environment and is transferred to your deployment preparation infrastructure. However, the bundle's payload/content is already signed and will not be changed anymore beyond this point!
Create and Deploy Device Keys
In this section, we simulate the factory process of creating and deploying asymmetric per-device keys/certificates that will be used for encryption and decryption.
The keys and certificates used for encryption do not need to have any relation to the keys and certificates used for the signing process.
To encrypt the bundle for a number of recipients, we typically need to have access to our device management system, which stores the devices' public recipient certificates. All steps described here are normally done before bringing your devices in the field.
In general, it is up to you and your use case how finely-grained you need to encrypt your bundle. Here we assume that each device has its own individual key. This allows revoking a device's key if it should be compromised. Future bundles will then be encrypted for all but the compromised key. See more about use cases and workflows in the RAUC documentation here
While RAUC potentially allows encrypting a bundle to a large number of recipients, this minimal example will use only a few recipients for the sake of simplicity.
Let's create some ECC key pairs with certificates first:
$ mkdir -p keys/private $ for i in $(seq 0 4); do > openssl ecparam -name prime256v1 -genkey -noout -out keys/private/private-key-$i.pem > openssl ec -in keys/private/private-key-$i.pem -pubout -out keys/public-key-$i.pem > openssl req -new -x509 -key keys/private/private-key-$i.pem -out keys/recipient-cert-$i.pem -days 365250 -subj "/O=RAUC/CN=RAUC Test $i" > done
This will generate you 5 individual encryption key pairs in the keys/ directory where the sensitive private keys are stored in the keys/private/ directory.
We use a single key pair as decryption keys for our demo target, the others could be used for other targets. Simply copy these key pairs to the devices' root filesystems (note that the final file names must match those we configured in system.conf):
$ scp -P 2222 keys/private/private-key-0.pem root@localhost:/etc/rauc/crypt-key.pem $ scp -P 2222 keys/recipient-cert-0.pem root@localhost:/etc/rauc/crypt-cert.pem
Your target's rootfs /etc/rauc directory should now look as follows:
root@qemux86-64:~# ls -1 /etc/rauc ca.cert.pem crypt-cert.pem crypt-key.pem system.conf
Your target is now fully capable of decrypting and installing RAUC bundles.
Encrypt Bundle For Our Recipients
Now that we have our target ready, in this section we will perform step 2 of the two-step bundle encryption: Creating a fully CMS-encrypted 'crypt' bundle for our target and other potential recipients.
To allow encrypting for a huge number of recipients, RAUC supports loading the recipient certificates as a concatenated list from a single file.
To create such, we simply run
$ cat keys/recipient-cert-0.pem keys/recipient-cert-2.pem keys/recipient-cert-4.pem > keys/recipients.pem
We intentionally omit recipient-cert-1.pem and recipient-cert-3.pem here for later purposes.
Now, we can encrypt our bundle using the new rauc encrypt command:
$ oe-run-native rauc-native rauc encrypt \ --keyring=example-ca/ca.cert.pem \ --to keys/recipients.pem tmp/deploy/images/qemux86-64/qemu-demo-bundle-qemux86-64.raucb \ encrypted.raucb Encrypted bundle written to encrypted.raucb
We could also specify each recipient cert separately by using --to multiple times, but this quickly becomes inconvenient, especially with larger numbers of recipients.
The resulting encrypted.raucb is now a fully encrypted crypt bundle.
A simple rauc info call (as already done above) should not show any information anymore.
$ oe-run-native rauc-native rauc info --keyring=example-ca/ca.cert.pem encrypted.raucb Encrypted bundle detected, but no decryption key given
Showing the bundle information is now only possible with one of the decryption keys given! We provide it and add the --dump-recipients argument to make the list of recipients of this bundle visible. Note that the Bundle format is now printed as [encrypted CMS].
$ oe-run-native rauc-native rauc info --keyring=example-ca/ca.cert.pem --key=keys/private/private-key-0.pem --dump-recipients encrypted.raucb Compatible: 'qemu86-64 demo platform' Version: '1.0' Description: 'qemu-demo-bundle version 1.0-r0' Build: '20220328213514' Hooks: '' Bundle Format: crypt [encrypted CMS] Crypt Key: '<hidden>' Verity Salt: 'e6800d82b6d37df11bae194e28abfbea53bd40e2ccd7ac1d85435a2ea47ab552' Verity Hash: '83ab98d255488e96c59b8e5fe122c70b6206b35c4b885f8ca3b8af1a95da7302' Verity Size: 512000 [...] 3 Recipients: 0 Issuer: O = RAUC, CN = RAUC Test 0 Serial: 0x3078C175A64B65BAADC9751FF4B52624D98CDE01 Algorithm: dhSinglePass-stdDH-sha1kdf-scheme 1 Issuer: O = RAUC, CN = RAUC Test 2 Serial: 0x61999A7BBC9B9FE3F05CB100B6E4D1899F699782 Algorithm: dhSinglePass-stdDH-sha1kdf-scheme 2 Issuer: O = RAUC, CN = RAUC Test 4 Serial: 0x620632AECA563A343A35BC53205021732C58AE19 Algorithm: dhSinglePass-stdDH-sha1kdf-scheme
Install Encrypted Bundle on the Target
Now, we have everything prepared to actually install the bundle on the target. This is exactly what we will do in this final section.
First, copy the bundle to the target.
$ scp -P 2222 encrypted.raucb root@localhost:/data/
We could also install the bundle directly over HTTPS by making use of RAUC's streaming capabilities, but since this requires us to set up an HTTP server first, we skip this for the sake of simplicity.
Then, on the target, simply run rauc install to install your encrypted bundle.
root@qemux86-64:~# rauc install /data/encrypted.raucb installing 0% Installing 0% Determining slot states 20% Determining slot states done. 20% Checking bundle 20% Verifying signature 40% Verifying signature done. 40% Checking bundle done. 40% Checking manifest contents 60% Checking manifest contents done. 60% Determining target install group 80% Determining target install group done. 80% Updating slots 80% Checking slot efi.0 85% Checking slot efi.0 done. 85% Copying image to efi.0 90% Copying image to efi.0 done. 90% Checking slot rootfs.1 95% Checking slot rootfs.1 done. 95% Copying image to rootfs.1 100% Copying image to rootfs.1 done. 100% Updating slots done. 100% Installing done. idle Installing `/data/encrypted.raucb` succeeded
Finally, to see that this is not all just for show but actually works, let's exchange the decryption key with one we did not sign the bundle for:
$ scp -P 2222 keys/private/private-key-3.pem root@localhost:/etc/rauc/crypt-key.pem $ scp -P 2222 keys/recipient-cert-3.pem root@localhost:/etc/rauc/crypt-cert.pem
root@qemux86-64:~# rauc install /data/encrypted.raucb installing 0% Installing 0% Determining slot states 20% Determining slot states done. 20% Checking bundle 40% Checking bundle failed. 100% Installing failed. LastError: Failed to decrypt bundle: Failed to decrypt CMS EnvelopedData: error:1C800066:Provider routines::cipher operation failed Installing `/data/encrypted.raucb` failed
As expected, the bundle decryption fails and thus the installation is aborted with an error.
You can now play around with encryption support on your own, integrate it into your setup (if you haven't done already while following the tutorial), discuss on our community channel IRC/Matrix or on GitHub, or contact Pengutronix for professional support.
RAUC is an update framework for safely deploying verified updates on your embedded Linux devices. It ensures atomicity of the update process to protect from sudden power outages, hardware failures, etc. So, why would one like to run RAUC on an emulated platform?
Unter dem Motto "Voll verteilt" finden die Chemnitzer Linux Tage auch 2022 im virtuellen Raum statt. Wie auch im letzten Jahr, könnt ihr uns in der bunten Pixelwelt des Workadventures treffen und auf einen Schnack über Linux, Open Source, oder neue Entwicklungen vorbei kommen.
Back in 2018, rauc-hawkbit-updater was started by Prevas A/S as a C/GLib port of our rauc-hawkbit Python prototype (also called RAUC hawkBit Client) that was mainly developed for showcases and to serve as a demonstration and evaluation platform for others.
Eingebettete Systeme und IoT-Geräte robust und sicher im Feld updaten zu können ist heute eine Kernanforderung jedes Produkts. Das Update-Framework RAUC ist die Basis für eine moderne und zukunftsfähige Lösung. In diesem Showcase zeigen wir die Grundprinzipien eines ausfallsicheren Update-Systems und wie Sie dieses mit Unterstützung von Pengutronix für Ihre Plattform realisieren können.
Nicht zuletzt da das Yocto Projekt aus der Embedded-Welt schon seit längerem nicht mehr weg zu denken ist, setzt auch Pengutronix bereits seit einigen Jahren neben PTXdist in diversen Projekten auch OpenEmbedded erfolgreich als Distributions-Buildtool ein.