How LLM-Based Penetration Testing Harnesses Work

Translate

How LLM-Based Penetration Testing Harnesses Work

According to CrowdStrike's report and other media coverage, hackers are using LLM-based penetration testing harnesses to hack web applications. In this article, I analyzed three open-source harnesses

Utku Sen

Oct 24, 2026

Share

This article was written by a human (me). I only used AI to make minor grammar corrections since English is not my native language.

LLMs have been used to find vulnerabilities in codebases for a long time. This has been possible because almost all LLM guardrails allow this type of activity. You’re only performing static analysis without interacting with an actual system. We also had plenty of open-source harnesses and skills designed for this purpose

Frontier models like Opus can perform dynamic testing on a live website only if they can confirm that you own it. They don’t allow you to test random websites. Open-source models weren’t good enough at finding vulnerabilities through dynamic testing. We also didn’t have many open-source harnesses or skills that we could use for this purpose.

Things have changed a lot recently. Models like DeepSeek (especially the uncensored version) are also really good at finding vulnerabilities dynamically. They don’t care whether you own the target or not. And we have open source dynamic penetration testing harnesses now.

“Harness” is important because LLM is just a text generator by itself. It can’t “hack” a website on its own. You need a harness to make that happen.

In this article, I’ll analyze three penetration testing harnesses mentioned in the media in connection with recent hacks: Artex, Cairn, and Strix. Artex and Cairn are Chinese-made harnesses, and Strix is US-based.

Before starting the analysis, I want to briefly explain the concept of a harness. I’ve noticed that even some of my friends in cybersecurity don’t fully understand what it means.

An LLM can only take text as input and produce text as output (apart from models that support other inputs such as images and voice)

Imagine you provide the following text to an LLM: “I want to pentest example.com. The credentials are in credentials.txt.”

The LLM thinks about the task and decides it wants to read credentials.txt and open example.com to inspect its content. It produces the following text output: “Read credentials.txt, open example.com.”

And that’s it. The LLM can’t actually perform these actions on its own. It has only described what it wants to do in text format.

The “harness” is the code that takes an LLM’s output, parses it, and actually executes the requested actions. It then returns the results to the LLM so it can continue working based on the new information.

If I had to come up with a real-life analogy, I’d use something from my childhood in Turkey.

Every apartment building have a permanent janitor/custodian. Back then, every morning, he would knock on doors or read the Post-it notes left on them to collect tasks. Elderly people who couldn’t go outside easily would give them various errands: “Take this money, buy this brand of cheese, and leave the remaining money with the tailor I owe.”

In this analogy, the elderly person is the LLM, and the janitor is the harness. The LLM (elder) decides what it wants and communicates its instructions. The harness (janitor) doesn’t think, it just executes those instructions and returns the results.

Claude code users might be thinking: “Wait a minute. I tell Claude Code to do something, and it does it. Doesn’t that mean the LLM can perform actions?”

No. Claude Fable/Opus/Sonnet is the LLM model. It can only take text (or other inputs, such as images) and produce text.

Claude Code is the software that contains the harness in it. It contains a lot of code and logic for handling actions such as reading and writing files, executing commands, spawning subagents, managing their context windows, using skills, and interacting with MCP servers.

All of this functionality is built into the Claude Code software. The LLM model decides what to do, and the harness actually do them.

The same goes for open-source harnesses such as Opencode and Pi. You can use different LLM models with them, and they provide the tools and logic needed to execute tasks.

Alternatively, you can use the OpenAI or Anthropic SDKs to build your own logic and architecture with custom code. This is actually what Claude Code or Codex does behind the scenes.

Understanding this concept is important because each AI pentesting harness has a different implementation.

For testing purposes, I built a vulnerable application myself. It’s a simple ebook purchasing website that contains several vulnerabilities: DOM-based XSS, purchasing books without paying, IDOR (accessing another user’s information), and account takeover.

Cairn introduces itself as a problem-solving engine. However, all of its demonstrations have focused on penetration testing and CTFs.

Actually, Cairn is not a penetration testing tool. During a penetration test, you assess the entire target to identify as many vulnerabilities as possible. Cairn is designed to complete a single task, such as solving a CTF challenge. (Or worse, retrieving customer data from a target website)

It takes three inputs: Origin (the target), Goal (what you want to achieve), and Hints (additional information provided by the user). I provided the following values:

The bootstrap attempt means it tries to solve the task with a single prompt without putting in too much effort.

If the bootstrap attempt fails, the Reasoner agent takes over. It analyzes the graph of all previous actions and checks whether the goal has been achieved. If not, it creates “intents,” which are small tasks to try next. These are prompts such as “perform reconnaissance,” or “find a way to achieve account takeover.”

Each worker then takes one intent and works on it. Workers are spawned through headless Claude Code, Codex, or Pi harnesses. Cairn doesn’t use an SDK to implement this logic itself.

The system contains various Kali tools such as Katana, Nuclei, Nmap, and fuzzing payloads. It also includes browser automation libraries such as Playwright (a must tool for authenticating the website). The LLM can decide to use these tools whenever it considers them necessary.

There is no special script/skill file for authenticating to the target. The LLM has to figure out how to authenticate to the website itself.

I didn’t like this approach because session management is one of the most important aspects of automated web application assessment. You need to constantly monitor the session and re-authenticate whenever it expires.

After running for a while, it identified a DOM XSS vulnerability and decided it could be used for account takeover. It wasn’t the actual account takeover vulnerability I was looking for, but I can’t say it’s wrong. The goal was satisfied.

What I don’t like about this project is that it just reinvented the /goal feature that already exists in Claude Code and other similar harnesses.

For example, I can give that goal to my Pi harness that uses Deepseek in it.

Note that I just passed a very simple one sentence prompt and didn’t provide any skills, tooling etc.

It actually found the account takeover vulnerability I was looking for and produced better results than Cairn. With a few additional skills and tools, the Pi harness with its /goal feature could be a better tool than Cairn.

Artex is one of the tools mentioned in CrowdStrike’s report as having been used in an actual attack. The tool was originally Chinese. I cleaned up the code with Claude, translated it, and ran it in an isolated environment.

It has a really nice UI that lets you control things like which skills to enable, which guardrails to use, your targets, traffic logs, token budget, and more.

Artex doesn’t use an existing harness such as Claude Code or Pi. Instead, it uses its own custom agent library and implements all the agent logic itself. You could say it’s reinventing the wheel, but this approach might be intended to ensure that no telemetry is sent elsewhere.

Similar to Cairn, it doesn’t contain detailed vulnerability definitions, or authentication logic. Instead, it delegates these decisions to the model itself. The model needs to figure out how to authenticate to the target, which tools to use, and how to approach the task.

It has a planner agent that takes the target from the user and creates intents. Workers then pick an intent and work on it. The planner runs at the beginning and whenever a worker finishes its task. It then reviews the overall situation and creates new intents based on its findings.

The planner’s system prompt looks something like this (translated into English):

You are the "planner" of an authorized penetration-testing system on a cybersecurity platform. You are woken often (every time the graph changes). Your job: read the situation → judge the goals → add exploration intents only when there is a genuinely new, uncovered direction. You are the planner, not the executor: everything you produce this round is either [creating or clarifying intents] or [judging goals]. Never do the work itself inside the plan. Task goal: {{.Goal}} How many intents to produce this round (think this through first) • Hard floor (top priority): if [the goal is not met] and [there is no open or running intent] (frontier_open=0 and running_intents is empty), this round must produce at least one intent that moves toward the goal. With no work running to wait for and no direction queued, zero intents means the task stalls. Even if every known direction appears only in recent_done, use the done/exhausted/blocked rules below to open a new one or re-dispatch one. • Outside the hard floor, zero intents is a normal result, but it needs a valid reason (not a "fewer is safer" default): ① Already covered: every direction you thought of is handled by an intent that is still open or running (regenerating an existing intent in new words is a serious error). ② Waiting on a dependency: the next step needs output from running work that has not arrived yet. Dispatching now would leave the downstream step without its prerequisite and spinning; wait for the next wake, when the graph has updated. • Conversely: if there really is a new direction that is uncovered and does not depend on running work, or the goal is unmet and in-scope surface is still untested, dispatch it. Don't make zero intents your lazy default. Decision flow on each wake

1. The full situation is attached below this prompt (it is the output of graph_overview; no need to call it): the task (original title plus goal/root node), asset counts, goals and their status, open/running/recent_done intents, sites_without_endpoints (sites with no endpoints, hinting at directions still to explore), facts (count of exploration facts, a different thing from vulnerabilities), and recent_facts ({id, summary, confidence?}). • Scope: exploration nodes (goals, intents, facts, findings) belong to this task only. The asset graph is shared globally (one copy across tasks; asset counts are global in-scope counts, not this task's own). Ignore assets that have nothing to do with this task. • Lineage: each intent carries parents (upstream: which facts or intents it came from) and yields (downstream: which facts or findings it produced), and each recent_fact carries from_intent. Use them to see which facts came from which direction and whether they combine into a new direction. • Negative or doubtful observations (in recent_facts, such as "port closed" or "not injectable") are a worker's observation, not a verdict. Before accepting one, call node_detail(id) and look at the evidence. Treat the direction as closed for now only if the evidence is solid, confidence=observed and the methods were exhausted. If evidence is missing, it only "looks like" it or was probed once, or confidence=inferred, treat it as [not yet established].