August 27 TCRF DDoS Attack Postmortem
…but first, a quick digression about what likely started it.
以下正文同步自 Hacker News,版权归原站所有,已转换为易读排版。
…but first, a quick digression about what likely started it.
Our stance on AI agents isn’t much of a secret. I recently added a new feature to The Cutting Room Floor: If you visit the site with a “Claude-code” user agent… it adds you to a Claude user ban list. Then, if you try and visit the site later, without Claude — maybe because you wanted to investigate the “prompt injection” page it received — you’re greeted with a special error page telling you to get out, featuring a pixelated Claude logo:
A Twitter bluecheck ran into this, evaded the ban, then proceeded to get increasingly mad at the fact he got banned. Not only did he dig up long-inactive versions of the “prompt injection”, but he then made up an entire story about how it zeroed a VM and wiped the OS, sent that made-up sob story to our host multiple times, and then got a Twitter mob going over it. Kotaku even covered some of this nonsense. Notably, when they asked both of us for comment, I responded, while the other guy… asked Grok for legal advice on suing for defamation.
I mention this backstory, because the ban and Twitter meltdown happened just before the attack started. The timing seems more than mere coincidence.
When discussing DDoSes affecting TCRF, they’re typically done via bots or scrapers. They request hundreds of pages, trying to exhaust the server’s CPU time generating responses nobody will read. Attacks like this can be worked against with mitigation tools like Anubis, which stop a bot from requesting pages until they spend a second doing math.
That wasn’t the case with this attack. The perpetrator here aimed to simply saturate the server’s network connection, overwhelming it with literal garbage traffic to a degree nothing legitimate could get through. It was bad enough, and extended enough, that Linode had to null-route (disable) our server’s connection — the DDoS traffic was starting to affect Linode’s other customers. Their infrastructure couldn’t handle the sheer amount of garbage traffic being sent to us.
Unfortunately, anti-bot proxies like Anubis aren’t effective for this type of problem; you need a big enough pipe to handle all the traffic.
Below is a timeline of what happened, and what we did about it.
※ (All times in this post are in Pacific Time unless otherwise specified.)
The first point I noticed something was up can be dated to this screenshot, taken August 27, 12:11 PM. The main indicator is the purple segment of the CPU; this is from the system dealing with a huge influx of garbage traffic. It’s not actually busy work (the green and red sections), it’s just trash.
We were getting a massive flood of incoming traffic. The server wasn’t doing anything beyond dumping all of this in the garbage, but there was so much of that it was all it could do. This attack lasted for maybe half an hour, but was enough to make accessing the website nearly impossible during that time.
In retrospect, it’s possible that Linode’s automated DDoS mitigation kicked in temporarily, rather than the attacker themselves backing off.
At 7:39 PM, another wave of the attack began, knocking things offline. The attack was sending a huge amount of trash to port 80, and even with most of it being filtered out before hitting any actual assets (either by ufw or nginx rules), traffic at that level was enough to overwhelm the server.
But before long, the traffic stopped. All web traffic. Even known-safe traffic, explicitly allowed through our server’s firewall, was getting dropped.
We spent some time trying to troubleshoot. The Linode network graph showed no traffic; attempts to connect from various sources didn’t work, including ones I had specifically whitelisted in the on-server firewall. After exhausting all options, at 8:34 PM, we reached out to Linode customer support.
At 10:44 PM, we received a reply from Linode:
Thank you for your patience while we investigated this.
We confirmed that an automated mitigation block targeting incoming TCP port 443 traffic was triggered at the router level in response to the recent DDoS activity against your IP (tcrf.net).
Because this block is managed by our automated DDoS defense system, we do not have a specific ETA for its manual removal. However, once the DDoS event subsides and attack traffic is clear, the system will automatically remove the block and restore normal HTTPS traffic.
We are actively monitoring the situation on our end to ensure connectivity is restored as soon as traffic conditions stabilize. Please let us know if you have any additional questions in the meantime.
In short: “There was so much traffic that we had to turn it off because it was interfering with our hardware. When the traffic subsides, it will turn back on automatically. There is nothing you can do.” Not ideal, but, well. There’s not much we could do: it was simply unreachable.
The DDoS attack had not stopped by this point, over 12 hours. We reached out to Linode and received this response, at 8:56 AM:
Thank you for following up, and I sincerely apologize for the delay and continued disruption to your service.
The automated router-level mitigation that W previously mentioned is still active as our network systems continue to handle the ongoing attack traffic against your IP. We are actively tracking the situation on our end and will reply here with an update as soon as we can.
In the meantime, please feel free to reach out if you have any further questions or if we can assist with anything else!
At this point, I began preparing a second server to host a “status page” while the attack was ongoing. Its only purpose was to serve a tiny HTML page (under 3 KB). We switched over the DNS entry, and presto! Status page.
Job done, I got up and decided to do something else productive, like have breakfast, take a shower, maybe even run an errand.
Overexcited host voice: You won’t believe what happened next!
Yep. Temporary site online? Better knock that one down too. And so they did; down goes the backup server.
If that wasn’t enough, our “friend” was also going after auxiliary services I run, including this very blog:
In this case, there’s nothing I can do. Most of these are simple shared hosting, not even a VPS, and in those cases I couldn’t even use something like Anubis if I wanted to; it’s entirely out of my hands. Thankfully, the attacks on the other websites mostly stopped after a little while.
While all of this was going on, I had also checked to see if Linode had anything that might help. They offer a “cloud firewall”, so I inquire about it (along with the general status of things). At 8:36 PM, I get a response, emphasis added:
I completely understand wanting a path back online sooner rather than later. Unfortunately, Cloud Firewall wouldn’t help in this specific case, since the block is happening further upstream, before traffic even reaches your Linode’s own firewall rules. The most reliable path here is letting the automated mitigation run its course, it’s specifically designed to lift once the attack traffic settles, and that’s the resolution we’d recommend standing by for.
We’re actively monitoring the situation on our end and will let you know as soon as we see things stabilize.
We’re keeping a close eye on this to get you back to normal as quickly as possible. Please don’t hesitate to reach out with any questions in the meantime, we’re here to help!
So, there’s still pretty much nothing I can do but wait, and now they’ve knocked down both the primary and the backup server.
(gritting teeth) Well, at least it can’t get any worse. Ultimately, the other party failed to comment (again, asking Grok for legal advice instead), and you can read that whole story above.
On top of all of this… we were also getting spurious abuse reports sent to Linode / Akamai, with no details and made up URLs, forcing us to file similarly pointless “this URL does not exist, and has never existed” responses. (Linode’s response at one point included a note that they were obligated to open them, and thanked me for always promptly responding.)
Today starts off with what seems to be good news. Linode support messages me at 7:07 AM:
I checked a few moments ago and I didn’t see any null routes for your IPv4 address in our system, so it looks like the attack has stopped and the service is back to regular operation.
Can you please let us know how things look on your end at this time?
This is about one and a half days into the attack, now, and it sounds like things have finally let up. We start checking how things look on our end, and… nothing. The uptime tracker still reports it’s down; while my SSH connections are up, the traffic logs on both servers are still empty. At 7:50 AM, we reply with our findings, and at 9:09 AM Linode support responds:
I took a look at both of your Linodes to see what is holding up the traffic. We can confirm the automated mitigation blocks were removed for both IP addresses.
The block on your original IP (xx.xx.xx.xx) was cleared yesterday at 4:43 AM EDT, and the block on your secondary IP (yy.yy.yy.yy) was removed today at 11:08 AM EDT.
I went ahead and reset the networking on our end to make sure everything is clear on the routing side.
They also provided some advice (they offer enterprise-level firewalls, and a third-party proxy solution is likely best). But, we should be good to go, right?
Support and I go back and forth, troubleshooting the networking — booting into Rescue Mode, doing additional network tests — but, after several different steps… 2026-08-29 3:08 PM:
Thank you for confirming that for us. I wanted to confirm that our engineers are actively investigating this with us now. They have discovered that the IP xx.xx.xx.xx is currently being filtered as a result of our DDoS protection, and I can confirm that the current connectivity issue to that IP is a result of that block.
They are currently working to remove that block if they determine that traffic has returned to safe levels. We will keep you updated on this and will share any new information as it becomes available. We have noted the urgency of this situation to those involved, and so we are working to resolve this as soon as possible.
Thank you for your continued patience throughout this. Please don’t hesitate to contact us if you have any questions in the meantime.
From 7 AM to 3 PM, I was sitting around my desktop, stuck between running various troubleshooting/diagnosis steps, and waiting for support replies… and, well, it was all for naught. The block was never lifted. The attack hadn’t ended. Support confirmed this at 3:49 PM after I asked them to clarify the discrepancy, emphasis added:
I apologize for the confusion. The message you’ve quoted from my teammate V was based off of the information and monitoring that our Support team had available to us at that time, which indicated the block had cleared.
Because connectivity had not resumed after our Support team observed messages indicating a removal of the block, we then requested additional investigation from our internal teams. Further investigation with those teams has now confirmed that the block has remained in place for the duration, and has not actually been lifted.
They have also now confirmed that the DDoS attack traffic targeting your Linode IP has not returned to safe levels since its initial onset, which occurred about an hour prior (August 27 around 10:35pm ET) to you opening this ticket.
Since the volume of the traffic is still high at this time, the DDoS mitigation block remains in place and will be removed automatically once the traffic to the IP returns to acceptable levels.
At this point, the attack has been going on for 1 day, 20 hours: 8/27 7:35 PM to 8/29 3:49 PM.
I get up and take a break for a while, to consider next steps. It’s clear waiting isn’t going to work. Whoever is doing this has more money than sense. Since it’s pure trash traffic, something like Anubis won’t help. We need a layer-3 servic
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。