What a Terrible Day to Have a Computer

Today* was fueled by caffeine and adrenaline.

If you're any sort of system administrator, or know any system administrators, you know what I'm talking about.

If you aren't and don't, the short version is that multiple pieces of software that form the foundation of large parts of the web were announced to have security vulnerabilities, and there was a mad dash to get things patched before (more) hell broke loose. It started yesterday, escalated several times, and is probably not actually over.

The unavoidable question is: so, is this was our lives are just like now? A life of running fire drills?

To start at the ending, my answers are (and please, realize that these are coming at the end of a long, over-stimulated day, and I'm talking about stuff that's still ongoing, for which we definitely don't have the complete story):

Now: I'm not a professional system administrator: I have been in the past, but now I'm a professor. I know quite a fair bit about computer and network security, including doing research in the area, but I'm not an infosec professional. I still play a small but significant role in administering my research group's computing infrastructure, as well as the testbed infrastructures we run for computer scientists, which have tens of thousands of users. So, take this piece with these grains of salt.

Fundamentally, I like administering computer systems - it's what drew me to computing research in the first place, and it's why I administer a bunch of stuff at home that I really would be, objectively speaking, better off not administering.

I also like working on teams with smart, competent, resourceful people, which is why I don't just computer at home by myself, I work with other faculty, with research staff, with students.

Administering computer systems is a social activity.

Here's the first thing that has gone horribly wrong in computer security recently.

There are certain secret numbers that, if you know them, you can use to impersonate secure connections to just about any server on the Internet. It may (or will absolutely not, if you are familiar with web security) surprise you to learn that this is actually the normal state of affairs, and it's by design: we, collectively, trust a certain set of organizations, called Certificate Authorities (CAs), to create and safeguard these numbers - you can prove that you know the number without revealing what it is, and if you do this, we trust you to attest to the identity of things like websites. This is called Public Key Cryptography, and it's extremely cool.

This is not the problem (well, it kind of is, but that's a whole different story).

The problem is that yesterday, we learned that one of these secret number is, and has been for about a year: 4.

Yes, technically 4 is just as random as any other integer, but this does seem bad, right?

The thing is that these numbers are supposed to be very very long, such that guessing them is infeasible. You can guess 4. I can guess 4. Anyone can guess 4.

I don't think we know the full story yet, but what it looks like is that one of the CAs that almost all of us trust generated some random secret numbers and threw away almost all of the digits, leaving them with secrets like 4, 56, and 155. They've been using them since early 2025. Why didn't someone notice this until now? I think it mostly boils down to (a) the fact that it's not as obvious to notice as my simplistic presentation here makes it sound, and (b) generally speaking, this is not something people are usually looking for (it probably will be now!)

The scramble yesterday was to stop trusting those numbers to secure websites, and to reduce our trust in the CA that "created" them, eMudhra. Did you know you trusted eMudhra? Well, you did.

This kind of situation has been planned for, and there systems in place that help mitigate it in some settings (primarily web browsers). For example, there are public lists maintained of numbers you shouldn't trust anymore: you can "revoke" the number 4, and some (but by no means all) clients will learn about it eventually. It's bad when these secrets are compromised, it's a legitimate major security problem, but things like this have happened before, and they're going to happen again, and there is an (imperfect) system in place to help make it not quite so disastrous.

Thus, for most of us, yesterday afternoon was largely a matter of making sure that software was generally up to date, and having a good laugh at memes about the number four.

Later in the day, the PHP foundation, which is responsible for the PHP language that runs a large part of the web (estimates on how much vary) announced that they were going to drop a patch for a security vulnerability that affected every single version of PHP going back decades.

Under a common way of scoring the severity of vulnerabilities, this one got a 9.2. Out of 10.

If you don't work in security, you might wonder: why tell us that they were going to release a patch, instead of just releasing it?

Well, once you release the patch, people can start trying to figure out what it fixed. And once they do that, well, they can start developing exploits, and using them.

And this is where, however many words I am into this post, I finally say: "AI"

Based on what we think we know now about the vulnerability (-ies, plural, as it turns out), they are somewhat complex to exploit. A few years ago, we would have probably assumed that it was going to take a while to make an exploit. In fact, that's exactly what happened: a few years ago, apparently, the potential for this vulnerability was noticed, but people didn't, at the time, figure out how to exploit it, so they considered it hypothetical and didn't prioritize fixing it. Recently, it was re-discovered, and an exploit found, with the assistance of an LLM.

But today, the assumption is: once the patch was out there, people would use LLMs to develop exploits, and this would happen fast.

The current state of the world is that bots scraping web pages to collect training data for LLMs are ubiquitous and relentless. My research group has had to battle them many times in the last few months, and we haven't "won," we've just kept the load on our servers to a level where they're still usable for their actual purpose. It seems very likely that some of these bots are run by people malicious enough to turn them into botnets that start exploiting this new vulnerability.

So, the current best practice is: let the OS vendors, big companies, clouds, etc. know, so they can start making packages for people to update to. Announce a time, give everyone a little while to get ready.

So, my research group got ready. We know which of our services use PHP - we wrote them! We divvied up responsibility for the major services between us. I threw together a quick scanner to check the rest of our IP space in case there was stuff we had forgotten (in fact, there was.)

Some of the machines we had to update were in datacenters. On campus. In our university's off-campus datacenter. On rooftops around our campus. In boxes attached the sides of buildings. In buses (yes, literally) - these, and the rooftop, and building-side nodes are part of our POWDER wireless testbed. Of course, it's 2026, and we visited none of those places - I was in my office, everyone else was at home.

Four of us were ready at 8 AM our time, including someone who was supposed to be leaving on vacation. Shortly before 8, we shut down many of the webservers, and firewalled off others.

As it turns out, the patch was huge. (It also came out something like 30 minutes early.) Looks like a re-write of the entire memory allocator, but I haven't dug too far yet. The vulnerability was, as expected, a Remote Code Execution (RCE) vuln - send the right traffic to a vulnerable server, it doesn't matter what application is running inside of PHP, you get to run something else, with the permissions of the webserver, of your choosing.

These are people I've worked with for 10 to 25 years, and it's great to work as a team. We co-ordinated over slack. We all know what each other know. When something goes wrong, we know who to ask. When something needs doing, we know who can do it. Everyone has a common goal, and overlapping, but diverse, expertise.

We build packages for our most of our FreeBSD servers ourselves, and we couldn't start until the patch was out. So a fair amount of time was just waiting for builds to complete. Some of the Linux distributions we use were "late" getting packages into their own repositories. More waiting. We installed new PHP packages, did very light testing, brought webservers back up. In one case, we had been planning to move a website to a new server anyway, but this forced our hand - there was some last minute databases syncing and whatnot to do. We had another server that simply cannot be updated, so we carefully cordoned it off so that it can only be reached by the hosts that actually need to contact it. While we were at it, we made sure to revoke the number four from all of our servers.

As the day went on, more of the group came online. Offered encouragement. Lent a hand. Like I said, this is a social process.

A lot of stuff was back up within 30-60 minutes. Some took a few hours.

While waiting for packages to build and stuff to restart, I Posted Through It on social media. It lightened my mood.

We let CloudLab and POWDER users know that everything was clear.

One of the problems we had is that we were on a version of PHP that reached end-of-life (EOL) a few months ago. So it didn't get patched, and we had to do a version update in addition to applying the patch. We knew this in advance, one of the reasons why it's good to give folks some warning.

Our code is, let us say, "vintage." It was originally written for PHP 3 (the world is on 8 now). Over the years, we've modernized it and ported it forward, and we were already on an earlier release in the 8.x series, so it wasn't a huge jump. But the code has some history, is what I'm saying.

Knowing this, I had done a quick check the night before, and it seemed that we were going to get a bunch of warnings about deprecated syntax, but nothing would break.

But I spent most of the rest of the day tracking down and fixing deprecation warnings. And installing the fixed code very carefully, hoping no users would notice - seems that they didn't.

In the late morning, GitLab released a patch for an RCE of their own. Completely unrelated (it's not written in PHP). This one didn't have a severity of 9.2 - it had a severity of 10.

We run a sizeable GitLab server, it has 700 users and 2.5k repositories - this is the main thing we have to defend against bots, so we know that they know it's there. (The bots mostly spend their time downloading every diff between every revision of several copies of ... the Linux kernel. Super rare stuff, guys. Never before has so much dumb shit been done in service of "smart" technologies.)

So, we had it down within minutes. Updated and back up 15 minutes later.

Somewhere in this mess, I don't know exactly when, NodeJS releases patches for their own high-severity RCE. All supported versions (probably old unsupported ones too.) Again, completely unrelated.

Luckily, we only run one NodeJS thing, and could just firewall it off from everything but the one machine that needs to use it.

(I do, however, run some stuff in my homelab that uses NodeJS. I just pulled it all down, only brought back up the bits that had updated their container images.)

I had a chat with a potential collaborator, went to class, talked with a friend, went home, and fell asleep.

I didn't manage to get my Real Job, giving my feedback