eMudhra ignored Mozilla Root investigation deadline; blames social media

(In reply to Dennis Jackson from comment #5)

The information in this comment is preliminary. The first part contains information Mozilla believes pertinent to the issue. The second part contains questions for eMudhra.

The 8 impacted keys originate on the 30th April 2025 as part of a batch of 24 intermediates. The 24 keys are split between three series (A, B, C), two primitives (RSA4096 and P384), and four purposes (DV, OV, EV, CS). The impacted keys are all the P384 keys for the B and C series. In May 2025 the A series were revoked and replaced by the A-1 series.

After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys.

I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates."

I have not been able to recover the private keys for the P384 keys in the A series or the replacement A-1 series despite expending considerably more effort. I have not yet started looking at the RSA keys.

Looking through related issues has turned up Bug 1970728 in which eMudhra generated CRLs for the 6 impacted TLS intermediates with invalid signatures. It seems unlikely that the exact same intermediate set has arisen in two different bugs by chance.

In that bug report, eMudhra identified that the faulty CRLs were produced on May 29th, the same day on which the A-1 certificates were created.

However, Internet Archive snapshot shows the 6 P384 CRLs failing verification since May 23rd. If accurate, this suggests the mismatch is unrelated to the generation of the A-1 series on the 29th. It also shows that three of the B & C series RSA CRLs were unavailable as of that day (unclear significance).

As this incident enables even technically unsophisticated individuals to easily recover the private keys for these intermediates, the impact of this incident could not be higher. Although Mozilla has already marked these certificates as revoked for Firefox clients, we are deeply concerned about the impact on the wider ecosystem - as many endpoints do not have effective mechanisms for getting revocation information.

Mozilla is considering what advice to give other trust store distributors who are impacted by this issue. Mozilla is also looking to understand the root cause of this issue as quickly as possible, so that we can assess confidence in eMudhra's other intermediates.

The hardware used for each of the issuers is in the attached file. These HSMs are FIPS 140-2 Level 3 compliant.

Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault.

We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)

The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted.

Some additional context - these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one).

We are still assessing the code to see except steps where this process went wrong.

The A and A-1 series were created in Entrust HSMs and were meant for deployment only in India.

The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.

There were human errors in multiple steps of the process that resulted in significant flaws in the key lifecycle integrity process for this KGC. We are now digging deeper for code flaws, process gaps, maker/checker controls specific to this issue. Over the next week, we will reverify every key ceremony we have done.