Caching, Bot Defense and a CDN

origin is accessible only via requests from the edge through bunny. Each request has a header set with a shared key, and Caddy rejects any requests to www without it. Anything live is excluded from edge caching: now playing, my contact form, and a few other APIs.Caching

I opted to have Go set cache lifetime per page with Cache-Control headers. There's a short default and longer ranges for pages that rarely change. I skipped bunny's Smart Cache (sometimes predictability is better than heuristics).Cache invalidation is handled using tags (much like I do with Redis): every response has a CDN-Tag; saving things in my admin is debounced for a few seconds and then purges the appropriate tag. The cache is also purged at startup.Redis caches database queries, bunny caches rendered HTML at the edge, and the page re-renders on every request that hits the origin.Database migrations run when the new server starts. They write to the database directly, don't invalidate tags (or go to jail or collect $100), and benefit from the startup check that dumps the public content cache.Blocking things

The important part! (Sort of.) I ran bunny's WAF in learning mode and tuned rules. If you block things outright, you block things you shouldn't (e.g. don't block contact form submissions in English). I maintain the list of crawlers I block in my CMS: each agent is a string tied to a path, and my robots.txt is generated, in part, from that relationship. These are now also synced into the CDN's rule set to block the same crawlers at the edge. bunny also provides their own block lists, and one of those lists blocked folks on Tor and VPNs (oops—that's set to challenge now).Analytics

My analytics are hand-rolled and minimal. I have country data displayed in a graph, and that was previously populated using a GeoIP lookup. bunny, conveniently, supplies a country header, and the analytics now use that. I tweaked the implementation to record visits on page load rather than exit (exit events don't fire as reliably). Crawlers get excluded altogether (yay blocking!).Reliability

bunny has a feature called Origin Shield that sets up an additional caching layer in front of my origin. Cache misses get consolidated into one point, and my origin server ends up seeing fewer requests. The added benefit (in addition to the performance) is that some of bunny's edge locations couldn't hit my origin (which is IPv4-only). Putting the shield behind an edge node that can reach the origin fixes that. The network topology is hostile and unknown and weird and unpredictable and magical.Breaking things

Feed readers rely on If-None-Match to avoid redownloading unchanged feeds. I've left mine uncached so my (minimal) feed analytics can count polls. bunny drops conditional-request headers headed for the origin even for pass-throughs it doesn't cache, which means every poll is a 200. If I can't see the 304, I can't handle it properly, and, for that, I am appropriately apologetic.Request coalescing: this will break live routes. It's super handy for near-simultaneous requests for the same relatively static thing where a single copy can be returned in a performant fashion. This is antithetical to the notion of things that return unique and/or live responses. I put in an explicit skip list for those routes.When I deploy, visitors will briefly see 502s. A new copy of the Go binaries gets copied to the origin server, the app restarts, and Caddy hasn't reloaded its config yet (said config gets copied over as part of the deploy). There's now a short retry for Caddy to help mitigate this.For my now playing display: a page cached on the edge could flicker when it rendered older album art next to a live stream. Stale vs. live data: bad experience. The new cover is decoded off-screen and swapped into the view.Done, mostly

Hand-maintained IP banning is gone, crawler lists are out of Caddy, GeoIP is gone, and most of that now lives at the edge. Oh, and I added an analytics chart of blocked bots. They've been fucking off more over time, which is nice. Things should be faster, too!

<div class=