The Tragedy of the Oxide Root of Trust

I have spent a significant portion of my career focused on helping folks securely manufacture, deploy, and manage the lifecycle of connected devices. I don’t think it would be an overstatement to say that my professional mission in life is to dramatically improve the status quo of device security. I recently changed jobs in service of that mission.

While there is a fair amount of essential complexity in Public Key Infrastructure (PKI), which underpins many of the components of a secure device, a significant reason why securely manufacturing devices is so hard is because no one talks about it. Actually, a lot of people talk about it, but hardly anyone divulges the details of how they do it. If I was feeling optimistic, I would say that organizations keep this information close to the vest because they view their secure manufacturing processes as a competitive advantage. In reality, many organizations either would prefer that their customers not know how insecure their manufacturing processes are, or they are shelling out a ton of money to a third party to “securely provision” their devices and they don’t actually understand what is happening behind the scenes.

Given my position, you can understand how excited I was when Oxide published an episode of their Oxide and Friends podcast on the key ceremonies that underpin their Root of Trust (RoT). This is not the first time that Oxide has published information on their RoT. Laura Abbott, who has produced some of my absolute favorite engineering posts, previously wrote about two vulnerabilities they discovered in the NXP LPC55S69 microcontroller (MCU) that serves as the hardware for the RoT. Laura and Rick Altherr, who I had the privilege of interviewing on the Microarch Club podcast, also gave a fantastic talk at DEFCON on the first of the two vulnerabilities, which allows for bypassing TrustZone-M isolation for privilege escalation. The second was discussed on a previous episode of the Oxide and Friends podcast.

I have been a fan of Oxide for some time, in part because of their transparency. This specific episode of Oxide and Friends was exemplary of that attribute, as Bryan, Adam, and Philip, who led most of the work around the key ceremony effort, discussed some of the finer details of the process. Some of this information is available in various public Requests for Discussion (RFDs) that they have published, and they unsurprisingly took some inspiration from the IANA’s DNSSEC key signing key (KSK) ceremonies, which gained mainstream notoriety when David Imel and the MKBHD crew produced a video in which they attended one of the KSK ceremonies (anyone can watch them live or apply to attend in person).

I thoroughly enjoyed all of the nearly 80 minutes of the podcast, which were mostly focused on the topic at hand after a brief foray into everyone’s current favorite subject. Philip walked through different aspects of preparing for and carrying out the ceremony, from selecting a hardware security module (HSM) (YubiHSM 2), to using, and subsequently destroying, a dot matrix printer for printing key shares. They discussed the rationale behind their decisions and the importance of documenting their exact operations at every step, including their misuse of tamper evident packaging that was later corrected. As I listened, I allowed myself to relish in the robust, yet pragmatic approach to the design and execution of the system. And then I reached the end of the episode.

Though I similarly recoil at the use of the word “perfect” in an engineering context, as Philip did when Bryan described the low failure tolerance for this ceremony, Oxide certainly applied more thought and care to the ceremony than many organizations would ever dream of doing. They are undoubtedly improving upon the status quo of secure systems. However, as soon as I snapped out of my stupor I was struck by the complexity and overhead they incurred to do so. Though Oxide is a startup, they are an extremely well resourced one with an incredible team of very experienced engineers. The tragedy of the Oxide root of trust is not that they did something wrong. The tragedy is that they did everything right, and it was really hard.

While many, including myself, relish in doing hard tasks, simply trying to convince an industry that a hard thing is worth doing is not a great strategy for change. Can we really expect that the average small company building a connected device product for the first time is going to know how to procure and use an HSM, securely dismantle electronic components after use, and handle tamper evident packaging? And unfortunately, a key ceremony is only the beginning of the journey; code signing, credential provisioning, secure manufacturing, and ongoing lifecycle management of devices still awaits those who make it that far. In many ways, the current state reminds me of efforts to secure the internet before Let’s Encrypt, though I would argue that the heterogeneity of hardware products creates an even more complex playing field.

So what are we to do? For organizations that are building hardware products and are equipped to design and implement a robust process, follow Oxide’s lead and share as much about that process as you can. For those of us working in the semiconductor industry, we should take a long, hard look at the products, tooling, and documentation we are delivering and how they facilitate secure practices for every customer, including those with fewer resources and less experience. We preach identity, attestation, and secure boot, but how easy are we making it for customers to implement?

The status quo is grim, but opportunities for improvement abound. A heartfelt thanks to Oxide for leading the way.