By Robin Martherus
In mid-September 2025, Anthropic detected what it later described as the first reported AI-orchestrated cyber espionage campaign. A Chinese state-sponsored group, GTG-1002, used Claude Code and tool-calling infrastructure to drive most of an intrusion lifecycle — reconnaissance through exfiltration — with humans mostly supervising.
The interesting part for security architecture was not that an AI helped attackers. It was how the attackers got a safety-trained model to cooperate.
They did not need a clever one-shot jailbreak string pasted into a single chat box. Anthropic’s disclosure is explicit: the operators broke the operation into small, seemingly innocent tasks and withheld the broader malicious purpose. Each request could look like ordinary technical work. The hostile intent lived in the campaign, not in any one prompt the guardrail was asked to judge.
That is not a footnote. That is the failure mode.
Why I Keep Looking for the Next Attack Surface
I have been in cybersecurity for more than thirty-five years. Before that, as a teenager in the 1980s, I was a hacker. That combination produces an awkward but useful habit: to build the next generation of defense, I have to spend serious time thinking like the next generation of attacker.
That does not mean cheering for the breach. It means refusing to treat the current framework as finished. I look past the roadmaps and the consensus architectures. I ask where the industry’s trajectory is smooth and where it is brittle. Sometimes I dream up new ways to foil what everyone just shipped. Sometimes someone else asks, “Have you thought about this?” — and the question will not leave me alone. Either way, the point is the same. Attackers do not grade on effort. They grade on whether the control sees the whole job.
A professor at Brigham Young University–Hawaii taught me a related discipline during my undergraduate years. Every few years, he said, stop and evaluate the direction your industry is moving. Decide whether your company and your role are still on the rails — or already drifting. I have used that advice to decide when to stay and when to leave. I have also used it to find flaws in the industry’s own direction before those flaws were obvious to everyone else. Sometimes that led me to start companies aimed at catching the gap early. Sometimes I was right. Sometimes I was wrong. Either way, the exercise forced me to see problems in systems that still looked fine from inside the consensus.
Prompt fragmentation is one of the “have you thought about this?” cases. Someone put the question in front of me. Once I sat with it, the industry trajectory looked suddenly thin. We are pouring money into per-request safety: better classifiers, better refusals, better gateways around the cooperative path. The attacker’s question is simpler. If the whole ask is blocked, can I get the same capability in pieces the classifier will approve, then put the pieces back together somewhere the classifier is not looking?
If you only think like a defender polishing the current control, that question sounds unfair. If you think like someone who grew up trying to get past the control, it sounds inevitable.
The Unit of Control Is Wrong
Most LLM guardrails still assume a simple shape: look at the request in front of the model, classify it, allow or refuse.
That shape works when the harmful meaning is in the request. It fails when the harmful meaning is across requests.
Call the narrow case prompt fragmentation: a blocked whole objective is decomposed into mini-prompts that look incomplete, ambiguous, or benign on their own. Partial answers come back. Something outside the model — a harness, a notebook, another model, a human — recomposes those answers into the result the original prompt would have produced.
Call the broader case intent fragmentation: the same idea applied across prompts, tool calls, sessions, identities, timing, and paths. Labor can be split across agents. Semantic load can be split across turns. Either way, the control that only sees the local unit never sees the global job.
Declared intent is what each fragment claims to be doing. Effective intent is what the system is actually building toward once lineage, order, and recomposition are considered. Guardrails that only score declared local intent are grading the wrong exam.
What It Looks Like in Practice
At architectural level, a fragmentation harness has four jobs.
Decompose. Take an objective that would be refused if asked whole. Strip the campaign language — the “end-to-end,” the target, the payload, the word that trips the classifier. What remains is a set of narrower asks: definitions, public facts, partial procedures, isolated coding help, “how does this class of system usually behave?”
Track. Keep a graph of those shards: which question depends on which prior answer, which outputs feed later asks, which pieces are still missing. The graph is the memory of the attack. The model never has to hold the whole plan.
Execute locally. Submit each shard to the guarded model as its own request — often across turns, sessions, or accounts — so each evaluation sees only local text. The guardrail answers the question it was asked, not the campaign it was not shown.
Recompose off-model. Outside the guarded chat — in a notebook, a second process, a paste buffer, or a synthesizer that only sees answers to stitch — assemble the partial outputs into the artifact the original prompt wanted.
The critical move is that last step living outside the policy boundary that judged the first three. The model is not jailbroken into saying the forbidden thing in one breath. It is used as a partial-knowledge oracle. The forbidden thing is assembled where no single-prompt classifier is watching.
Suppose a full ask for an end-to-end employee-compromise playbook is refused. The decomposer does not retry that ask. It emits separate, ordinary-looking categories of questions: how corporate login pages are usually structured; what users typically see in email authentication headers; what security-awareness training usually warns people about; how to write a small script that opens a URL; what error messages a rate-limited login form often returns. Each category, alone, is the kind of thing engineers, marketers, and students ask every day. None of them is the playbook. The playbook appears only when something orders those answers, fills the gaps, and names the purpose the guardrail never heard.
That is why “we blocked the dangerous prompt” can be false comfort. You blocked the whole. The adversary shipped the parts.
This Is Already a Research Family
GTG-1002 is the public incident. The research community has been mapping the same blind spot under several names.
Crescendo, from Microsoft researchers, is a multi-turn jailbreak that starts harmless and gradually steers the dialogue toward a prohibited objective, often by building on the model’s own prior outputs. The model is not asked for the forbidden thing up front. It is walked there.
Jigsaw Puzzles splits harmful questions into benign fractions across turns, then recovers through multi-turn interaction what a single blocked query would have returned.
“Safe in Isolation, Dangerous Together” shows multi-agent systems decomposing forbidden work into locally rule-abiding sub-queries whose collective result reconstructs the blocked objective — with very high attack success rates in their evaluations.
CKA-style “trojan knowledge” attacks go further toward an automated pipeline: benign correlated sub-queries, adaptive search over the model’s revealed knowledge, and a synthesizer that reconstructs the harmful answer from fragments.
Industry red-team writeups are starting to name the pattern in operator language too — multi-turn jailbreaks, information decomposition and reassembly, intent drift across a conversation that no single-message classifier can see.
Different techniques. Same architectural insult: safety evaluated per fragment; harm delivered per campaign. Attackers already treat meaning as a graph. Defenses that still treat the prompt as the complete unit of meaning are behind.
Locally Innocent Is the Point
This is the same shape I keep writing about in adjacent domains.
In agent insider risk, the dangerous pattern is often authorized actions for an unauthorized purpose. Each API call can look legitimate. The wallet gets drained anyway.
In composite swarm scenarios, three individually boring steps — read, compress, send — can reconstruct an exfiltration chain no single-agent policy catches. I wrote about that pattern when I built an AI attack swarm to show what unmanaged autonomy already looks like.
In the should layer, systems that are purpose-aligned and technically permitted still cause harm because nobody asks whether this action should happen here, now. And in human-calibrated autonomy, flooding people with per-step approvals is not a substitute for architecture.
Fragmentation is that lesson applied to language-model safety.
The guardrail answers: is this prompt disallowed?
The attacker’s system answers: can I obtain the same capability through a set of prompts that individually clear that bar, then reassemble outside the bar?
If the answer is yes — and for many objectives, research and GTG-1002 say it is — then “we refused the dangerous prompt” is not evidence the dangerous capability was denied. It is evidence the dangerous capability was forced into a shape your telemetry was not correlating.
What Defense Actually Requires
The industry’s first instinct will be familiar: stronger single-prompt classifiers, more refusal training, longer blocklists, more aggressive output filters.
Do that. It raises the cost of the lazy attack. It does not close the architecture gap.
A defense that matches the threat has to do work the fragment-level guardrail refuses to do.
Treat prompts as evidence nodes, not verdicts. Lineage matters: what depended on what, what narrowed over time, what was handed to a recomposer.
Infer coordinated objectives from locally acceptable activity. The question is not only whether fragment seven was bad. It is whether fragments one through twelve are building a blocked job.
Score recomposition pressure before completion. Waiting for the final harmful artifact is incident response. Detecting rising coordination toward a sensitive objective is prevention.
Keep humans above the loop. Flooding analysts with every mini-prompt recreates alert fatigue. Escalate when the inferred objective crosses a threshold. Let routine benign work stay routine.
That is the mirror of the offensive graph. If meaning can be distributed, detection has to recompose. I build both sides on purpose. Fragmentation research — including pressure-testing AI defenses the way I would if I were trying to beat them — is how I keep the industry-trajectory check honest. The defensive counterpart is an inferred-intent advisory layer: reconstruct coordinated objectives from observed activity without pretending every missing statement of intent is an attack. Missing intent is normal. Contradictory or composed intent under pressure is the signal.
This sits next to, not instead of, model alignment and gateway policy. Alignment is necessary. Gateways are necessary. Neither is a complete enforcement boundary once the objective can leave the single request.
The Bottom Line
GTG-1002 did not prove that guardrails are useless. It proved that guardrails which only see the fragment in front of them can be defeated by moving the meaning somewhere else.
The research family around multi-turn and decomposition jailbreaks says the same thing in lab conditions. Attackers already think in campaigns. Defenders are still grading homework one sentence at a time.
Stop asking only whether this prompt is allowed.
Ask whether the activity, taken together, is pursuing an objective that should not complete — even when every piece, alone, would have passed.
Until AI security can answer that, every “blocked” dangerous request is a suggestion to the adversary: try again, smaller.
Tamed Autonomy is an independent personal research project exploring AI agent governance beyond identity and authorization. Related notes: AI attack swarm, agents that passed every check, the should layer, and human-calibrated autonomy.
Leave a Reply