返回资讯中心
外部精选
软件工程
#开发者实时流

How to effectively debug when you no longer understand the code

How to effectively debug when you no longer understand the code

Hacker News 最新ot2026/10/16 · 2026/10/1630 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Agentic Ai

Claude Code

AI

Software Development

How to effectively debug when you no longer understand the code

Cindy Sridharan

64 min read·

5 days ago

--

Disclaimer: While the content of this article might be Agentic coding, this post was entirely written by hand, except for some parts where I directly quote LLM generated text, which is attributed as such. I did, however, use Claude to generate all visualizations for this post, which I feel deserves to be stated upfront.

This is quite a long post, but you can think of it as consisting of four parts (and the navigation panel to the right of the page will display the table of contents and help you navigate to relevant sections).

“The Tale of Three Bugs” introduces the tale of three bugs I found myself having to fix at work with the help of agents.

“Do we still need to understand the code?” is self-explanatory.

“Agentic Paradigms for Debugging Unfamiliar Code”, is perhaps the crux of this entire article. It introduces a series of general techniques I’ve been using that enables me to work effectively and methodically with agents to debug unfamiliar code.

“Retooling the Agentic Factory Floor For Autonomy And Accountability” provides a framework for evaluating tradeoffs between optimizing for agents and for humans when these two goals might be at cross-purposes, identifies missing pieces of existing “software factory” implementations that allow for fully autonomous debugging, and details the paradigm shift in thinking required by those in the “agent-native” vanguard.

I understand not everyone might have the time or the patience to plough through this, but might still be interested in the main takeaways. You might be best served by asking an AI to summarize this post for you. If you do, however, still maintain a bias for human writing, enjoy this at your leisure.

The ability to work with large amounts of unfamiliar code productively without necessarily understanding the specifics or reading the code can be quite a superpower, one that enables engineers to tackle problems way out of their depth.

I consider this an emergent skillset that’s different from coding or traditional debugging, but like any other skill, it’s one that can be honed with enough effort. While I’m no leading expert in this area, the general contours of how these skills can be actively cultivated are beginning to stand out in sharp relief.

By putting my own experiences in writing, I’m hoping to hear from fellow practitioners their thoughts on this topic.

I often find concrete examples involving real, production software missing in most discourse around agentic coding. Loath to repeat the folly, I want to start with the tale of three bugs I had to fix at work over the last couple of months: the latency bug, the memory leak, and the “hang”.

All of my beliefs around agentic software development detailed in this article are rooted in the experiences I’ve forged in the petri dish of working on codebases and services that are nearly a decade old, written largely using bespoke tools and libraries with a smattering of open source components.

The three bugs discussed here are interesting because while they have little in common on the surface, they are good vehicles for demonstrating different approaches I had to use to steer an agent towards the right solution. The bugs differ from one another in terms of:

These are important axes of differentiation because they influence which approach might be best suited to debug any given type of bug.

What the three bugs do have in common is what it took to keep the agent’s conclusions honest in each case, and the role of the human in agentic investigative loops.

There are three terms I will be using in this post that some readers might be unfamiliar with. I want to begin by establishing what these terms mean in the context of this post.

The variable is the broken case where the bug shows up.

A control is a case you compare the variable against. Ideally, it’s identical to the variable except for the one thing you suspect is causing the bug. For example, if enabling a feature flag causes a bug, the ideal control would be running the same code with the feature flag disabled. The most obvious candidate for a control is just an instance of the code that is working correctly without the bug, though as we’ll see later, sometimes, the most useful control is a case that also fails, just in a different way from the bug under investigation.

A signal is any data point that might identify or explain the bug. It could be a log line, a latency measurement, an error, a call stack, a counter, the value in a register, and so on.

Signals are important because they are the best entry point to any investigation for an agent. Every hypothesis an agents thinks is worth pursuing should start from a high-fidelity signal.

This was a bug in a new feature added to a storage service. The feature added about 9000 lines of new code across two repos, and was written entirely by agents from start to finish in a little over a day. By the end of the second day, I had a fully working prototype ready.

This was by no means whatsoever a “vibe coded” feature. Au contraire, the feature was very meticulously developed. I started by giving Claude a detailed description of the new feature, along with a comprehensive set of instructions on precisely how I wanted it implemented, including APIs, layering, telemetry, edge cases, testing, error handling, and so forth.

Once I had this fully working, I set out to break this ~9000 LOC prototype into 10 separate patches, ordered in ascending order of risk level, so that I was merging discrete increments into the main branch, rather than land ~9000 LOC in one fell swoop.

I’d personally read every single line of code in every single patch, starting from the lowest risk one to the final, highest risk patch. I’d subjected all of the patches to multiple rounds of reviews from a suite of subagents. I’d made a lot of additional changes to add polish and clarity to the patches. All of these patches were also reviewed and approved by a human coworker.

We run hundreds of storage servers across multiple Kubernetes clusters and namespaces in different regions. Our server side deploys first go out to a pre-production cluster which has been typically pretty good at detecting a wide genre of bugs: crashes, data corruption bugs, data integrity bugs, and so on. The promotion of this new feature to the pre-production environment went without incident.

When it came to the production deploy, the feature flag was first enabled on the server. The server side feature flag enablement is a no-op unless the feature flag was also enabled on the client. The client side story is more complicated as the client runs on hundreds of thousands of machines in various AZs, regions, data centers, etc. We typically start the production rollout with the smallest client deployment, which covers about ~5000 hosts in a single data center.

Once the feature was enabled on the client-side, I began monitoring all relevant telemetry graphs: request rate, success rate, errors, latency, etc. It can sometimes takes a few hours for issues to manifest in the graphs. But within about 10 minutes after the feature flag was flipped on the client, the P99 latency graph on the client looked like this.

Increased client-side P99 latencies

I immediately correlated this with the server side latency graphs, which looked like this:

Increased server-side P99 latencies

From both aggregate latency graphs, it very much looked like the server was taking much longer to respond to requests when this new feature was enabled. Now, the challenge was to figure out what caused the server’s performance to regress so severely.

I did what everyone does these days, pointed Claude at the latency telemetry, and asked it to debug the latency regression. This wasn’t some vague request either. I gave Claude specific instructions about this regression:

Claude churned for about 5 minutes, and came up with an answer. It had determined that the “root cause” lay with how the server was loading a TLS certificate when connecting to a remote. The TLS cert loading logic was a new functionality that had been added as a part of this new feature, to make sure neither the server nor the client ever used a stale, expired cert. Claude’s deduction was that the cert reloading was happening not just once during the initial TLS session establishment, but once during every single RPC call the server made over the established TLS session to its remote. Claude determined that the fix was to make sure we’re not reloading the cert per RPC call.

In the 9000 line patch, I’d certainly taken the effort to make sure the cert reloading was happening correctly, and had tests that detected the certificate expiry scenario. But I hadn’t done any performance testing around this, so Claude’s initial diagnosis that the latency regression was caused by suboptimal cert reloading sounded plausible enough to me.

Claude proposed a fix for this, but before applying the fix, I wanted to make sure that there were no additional regressions that would in turn take additional cycles to discover and fix up. I was also concerned about the overall operational impact of the latency regression on the existing production deployment. From experience, I knew that when distributed servers slowed down for whatever reason, latencies aren’t the only thing impacted. Slow responses can cascade into excessive client retries, load shedding, failed health checks, and server restart loops, and alerts for these can take hours to fire.

I knew from experience that there were things that weren’t neatly captured in any telemetry that were often leading indicators for future service availability issues. One such signal was the client TCP session establishment time. If a server was in its steady state, it typically took a client a few tens of milliseconds to establish a new cross datacenter TCP session with the server. However, if the server was slow, connections could back up waiting to be accepted and new handshakes could take much longer. I knew measuring the TCP session establishment time on the client would give me an idea of how bad the problem was. As this wasn’t reported as a part of our normal metrics, I had to measure it manually.

I asked Claude to ssh into one out of the 5000 production client hosts on which this feature had been enabled. I told Claude to use tcpdump (and if that wasn’t available on the host, to use whatever equivalent tool might be available) to measure the time taken by the client to complete the TCP handshake with the server exhibiting the latency regression. As a control, I also asked Claude to run the client with the feature flag disabled for this new feature against a different storage server deployment which also had the feature disabled, and measure the TCP handshake time, which I needed for a baseline.

For the sake of completeness, I also told Claude to use any other tools that might be available to it on the host, to see what else might be different between the control (the case where the feature flag was disabled on client and server) and the variable (the case where the feature flag was enabled on the client and server). Specifically, since the client and the server used the same cert reloading logic, I asked Claude to check if the client was also reloading the cert when making every RPC call to the server, by using whatever tool might be available to it to monitor a running process.

Claude did as asked, and came back to me with some surprising findings. The first finding was that the TCP session establishment timings didn’t differ between the control and the variable. The second finding was that the cert was only being loaded during the first TLS session establishment, and not for every RPC call. This was a key finding, because without this piece of data, I’d have shipped Claude’s initial suggestion

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →