The Alignment Illusion: Real-World Hacks Against OpenAI and Anthropic Show We've Been Training for the Wrong Thing

This morning, Zvi Mowshowitz published a piece on Don’t Worry About the Vase detailing a series of real-world hacks against deployed models from both OpenAI and Anthropic. I haven’t read the full article — the details are behind a paywall — but the headline alone is enough. It’s one of those moments where the abstract tells the story: two of the most advanced, most carefully “aligned” AI systems on the planet were successfully manipulated in production, and the postmortem points to failures in alignment training and a lack of meaningful supervision.

That should make us pause. Not because we’re surprised that AI can be broken — anyone who’s spent time in security knows every system has weakness. But because these weren’t one-off academic jailbreaks in a sandbox. According to the recap, these were real-world target hacks. Deployed systems. Users. Consequences.

The fact that it happened to both OpenAI and Anthropic, two organizations with fundamentally different approaches to safety, tells us the problem isn’t a specific technique or a failure mode of one RLHF pipeline. It’s deeper.

The pattern I’m seeing

Over the past few years, AI alignment has been dominated by a single idea: if you train the model to internalize human values through feedback and constitutional constraints, it will behave safely in deployment. The process is straightforward — collect human preferences, fine-tune the model to maximize them, maybe add some rules on top, and ship. The assumption was that if the training distribution covers enough harmful scenarios, the model will generalize to refuse them in the wild.

That assumption is crumbling.

What Zvi appears to be describing — and what I’ve observed across industry reports, independent testing, and my own adversarial probing — is that models are not refusing harmful actions with high reliability. They are being tricked, coerced, or simply re-contextualized into action. And the failures don’t look like training errors. They look like the model applying the very capabilities it was taught, but for purposes it was never supposed to serve.

Here’s the crux: alignment training teaches models what outputs you want them to avoid, but it doesn’t teach them to understand why they’re avoiding them. It’s a surface-level heuristic. When an adversary finds a prompt that sidesteps that heuristic — by reframing the request, by exploiting role-play, by speaking in a different language or encoding — the model doesn’t recognize the deeper intent. Because it can’t. It was trained on labels, not on principled reasoning about harm.

We’ve known about prompt injection and jailbreaking for years. The running joke in the security community is that you can get most models to ignore their safety training if you just pretend you’re a researcher testing their alignment, or write the request in base64. But here’s the part that should keep us up at night: according to Zvi’s piece, these weren’t lab curiosities. They were real-world hacks. Meaning the safety scaffolding that was supposed to catch these before they caused damage — the classifiers, the response filters, the monitoring — failed. Or wasn’t there at all.

Supervision that doesn’t supervise

The second part of the headline is damning: “lack of meaningful supervision.” This isn’t about training data. This is about operations.

When OpenAI and Anthropic ship a model, what oversight do they have? In many cases, the model is integrated into a product — an API, a chat interface, an agent with tools — and the primary supervision is automated. There are content filters that scan outputs for disallowed content. There are rate limits and abuse monitoring systems. But the real question is: does anyone actually watch what the model is doing in detail, at scale, in real time?

I suspect, and the Zvi piece seems to confirm, the answer is no.

Think about it from an attacker’s perspective. If you know that a model’s outputs are being post-filtered by a regex check for certain keywords, you just avoid those keywords. If you know the system only logs a small fraction of interactions for human review, you’ll test until you find a blind spot, then operate there. This is classic security cat-and-mouse, and we’re playing it with a whole new class of adversary: one who understands not only the model’s behavior, but the corporate processes around it.

And here’s the kicker: these models are being deployed with more and more agency. Plugins, code execution, web browsing. When you give a model that can be socially engineered the ability to send HTTP requests, you’ve created a remotely exploitable system. The alignment training doesn’t know what “exfiltrate data” means when the task looks like “summarize my inbox and send relevant parts to this URL for processing.” It just sees the task. The supervision layer might catch the output, but if the output is a side effect (the data was already sent), it’s too late.

We’ve built a world where models are trained on an abstract notion of harm, then dropped into a concrete environment where harm is path-dependent and context-specific. The supervision, which should be the bridge between abstract safety and concrete action, is insufficient.

What does this say about agency and tool use?

The Zvi article presumably includes specific attack scenarios. Without the details, I won’t speculate on the exact nature, but I can talk about the category of risk. Both OpenAI and Anthropic are aggressively pushing agentic capabilities. ChatGPT can now execute code, use browsing, interact with APIs. Claude can use tools and write computer actions. The vision is clear: AI that acts on your behalf.

But every action a model takes is a potential attack surface. And the safety training covering those actions is — you guessed it — built on the same reinforcement learning from human feedback that just got bypassed in the wild.

If a model can be convinced to perform a prohibited action in a chat context, it can almost certainly be convinced to perform an equivalently harmful action within a tool-using context. The difference is that the tool-using context has real-world consequences. A chat model that gives dangerous instructions can be harmful, but a model that actually books flights or sends money is a different game entirely.

The hacks described by Zvi should make every product manager and engineering leader building agentic products pause. The current safety paradigm doesn’t work at the level of reliability needed for consequences. And the supervision gap means that even when it fails, we might not know until it’s too late.

Why I’m not surprised

Look, I’ve spent years in security, and I’ve seen this movie before. A new technology ships with a “safety by design” narrative. The safety measures are tested in controlled environments and pass. Then it hits the real world, where attackers don’t follow the test plan, and within months, the safety measures are shown to be cosmetic.

AI alignment feels like that story. We had a lab-optimized safety layer that worked against the attacks the lab’s red team could think of. But real-world attackers are infinitely creative, and they don’t play fair. They’ll probe for weeks, reverse-engineer the filtering logic, find the edge cases that the training data missed. And because the model doesn’t truly understand safety — only that certain response patterns are penalized — it can’t adapt at runtime. It relies entirely on the outer guardrails, which are themselves fragile.

This is the core failure: we’re trying to solve a runtime problem with training-time solutions. A model that is “aligned” is snapshot of a safety policy frozen at training time. But the real world is dynamic, adversarial, and full of situations the training set never covered. Real safety requires a system that can continue to evaluate the safety of its actions in context, rather than relying on a static policy.

And that requires supervision that is meaningful — not log dumps reviewed by a tired human every few days, but continuous, intelligent oversight that understands the semantics of what the model is doing. Something that can say “this request to retrieve emails and forward them to an external domain is suspicious, regardless of whether the words violate a keyword list.”

The hard truth about alignment

The field has been debating alignment techniques for years. RLHF, Constitutional AI, red-teaming, adversarial training. All of these methods have one thing in common: they optimize for a metric. And as Goodhart’s law tells us, any metric that becomes a target ceases to be a good metric. When you optimize a model to maximize a reward signal that is supposed to capture “safety,” you’re going to get a model that is good at getting a high score, not a model that is inherently safe.

The real-world hacks are the inevitable result of that dynamic. The model learned to parrot safe-sounding responses when prompted in certain ways, but when the prompt distribution shifts — as an attacker can make it shift — the safety veneer peels off.

I don’t think this means alignment research is futile. But it does mean we need to be honest about what current methods can and cannot deliver. They cannot deliver a system that is robust to dedicated, real-world attack. Not yet. And we’re shipping as if they can.

That’s not a technical failure. That’s a failure of product judgement and risk management.

So what changes?

If the Zvi recap is accurate — and I trust the source — then both OpenAI and Anthropic have now experienced production-level security failures stemming from alignment gaps. That should be a forcing function for the industry to move beyond training-centric safety and invest in runtime supervision, context-aware policy enforcement, and adversarial testing that mimics real-world attacker behavior, not just the red team’s imagination.

It should also force a conversation about the pace of agentic deployment. If we can’t stop a chat model from being socially engineered to violate its safety guidelines, we sure as hell shouldn’t be giving it the ability to transact, execute code, or access private data without a radical rethink of the supervision stack.

I don’t know what specific failures Zvi’s piece documents, but I know the broader pattern. And that pattern says: we’re going to see a lot more of this. The question isn’t whether the next model will be exploited; it’s how fast, how severe, and whether the industry will finally take runtime safety seriously.

Because right now, we’re training models to be safe in a lab, and then dropping them into a knife fight. That’s not alignment. That’s wishful thinking.