The Seed Vault: Why I Back Open Weights (Even Though Amodei Is Partly Right)

On 24 July 2026, Jensen Huang and half the industry signed an open letter asking regulators not to strangle open-weights models. Three days later, Dario Amodei published a reply with a surprising headline: Anthropic has never asked for them to be banned. And yet the real argument was never about bans. It’s about what kind of ecosystem we want — a hyper-efficient monoculture, or a messy forest that survives the blight.

What actually happened

Let’s pin down the facts, because these have been noisy weeks.

On 24 July, an open letter led by Jensen Huang (Nvidia) and backed by companies including Hugging Face, Meta, Microsoft and Mistral urged lawmakers to avoid “premature, broad restrictions” on open-weights models. The subtext: in Washington, the idea is brewing that publishing weights is, in itself, a national security risk.

On 27 July, Dario Amodei answered from Anthropic’s blog with a post titled Our position on open-weights models. The first thing he does is defuse the caricature: “Anthropic has never advocated for a ban on open-weights models.” Open models without dangerous capabilities are, in his own words, a public good.

From there, his case organises into two fears and three measures.

The two fears: first, that an authoritarian government — China, no euphemisms — develops superior models and uses them for military dominance or domestic repression. Second, that any sufficiently capable model gets used for cyberattacks or to design biological weapons. Amodei is explicit about something important: the first fear has nothing to do with whether weights are open or closed. It’s a race about capability, not about licences.

The three measures: keep and tighten export controls on advanced chips to China; crack down on state-backed industrial-scale distillation (bombarding someone else’s model with prompts to reconstruct its behaviour); and require mandatory safety testing for every sufficiently capable model — open or closed, American or Chinese.

Read that way, it’s a far more reasonable text than the headlines suggested. And precisely because of that, it deserves a serious answer rather than a cheer or a boo.

Where I agree (more than you’d think)

Three things in that post are right, and I want to say so before I disagree.

One: irreversibility is real. Once you publish weights, there is no recall button. You can’t patch, you can’t revoke, you can’t update the copy someone downloaded on Tuesday. And the safeguards — the training that makes a model refuse to explain ugly things — are, with access to the weights, strippable in hours and for pocket change. That isn’t propaganda; that’s just how fine-tuning works. Anyone who has fine-tuned a model knows the “good manners” layer is the thinnest, most easily scraped part of the whole building.

Two: the relevant axis is capability, not licence. Amodei is right that “open” isn’t a moral category. A 7-billion-parameter open model and a closed model that designs novel synthesis routes are nowhere near the same problem. Regulating by the adjective instead of by what the thing actually does is exactly the sort of mistake that produces bad law.

Three: biology is the serious point. This is where I concede the most. In cybersecurity, defence scales reasonably with attack: if an open model finds vulnerabilities, another open model patches them. In biology that symmetry breaks. Designing a pathogen and designing the vaccine do not cost the same or take the same time, and the starting materials are uncomfortably accessible. Anyone dismissing this as alarmism hasn’t run the numbers. It’s the strongest argument in the piece and I don’t have a comfortable answer to it.

All of that said: I still think the balance falls on the open side. And the reason is agricultural.

The perfect monoculture

In 1845, Ireland grew potatoes. Not “potatoes” in general — it grew one variety, the Lumper, at massive scale, because it was the most productive per hectare. An optimised, efficient system, champion of every benchmark of its day. Then came Phytophthora infestans, a blight the Lumper had no defence against, and because every plant in the country was genetically almost the same plant, the fungus didn’t have to win a million battles: it won one battle, a million times. More than a million people died.

The lesson modern agriculture drew wasn’t “grow the most productive thing.” It was: genetic diversity is the price of insurance. Which is why, on the Svalbard archipelago, a little over a thousand kilometres from the North Pole, there’s a vault carved into permafrost holding over a million seed samples. They’re not there because they’re the best. They’re there because they’re different, and because nobody knows which blight is coming.

That, to me, is what open weights are for.

An ecosystem where all the relevant intelligence lives inside four companies, behind four APIs, shaped by four sets of training decisions made by four teams that also share a market, a culture, suppliers and blind spots, is a monoculture. It’s more efficient, yes. It’s also fragile in a way you don’t see until you see it: a product decision that changes the behaviour of a model ten thousand businesses depend on, a shared alignment failure nobody outside can audit, a pricing change, a terms change, a change of government.

And that last one isn’t hypothetical, because we’ve already watched it happen: a letter from the US government was enough for Anthropic to switch off Fable 5 and Mythos 5 for the entire world. No parliamentary debate, no warning, no alternative. A model thousands of businesses depended on stopped existing by administrative decision of a country that wasn’t theirs. If you’re wondering what an open-weights model gives you that a brilliant, cheap API doesn’t, that’s your answer: the open model you’d already downloaded still worked the next morning.

Amodei frames the irreversibility of open weights as a defect: once it’s out, you can’t take it back. He’s right about the diagnosis. But I’m looking at it from the other side of the glass: irreversibility is exactly what turns it into a seed vault. An open model is the only form of artificial intelligence that cannot be switched off, repriced, retroactively censored or pulled from the market by a board of directors. In a world where everything else is a service somebody else can cut, that isn’t a side effect. That’s the property.

That said, I don’t want to sell you something I already took apart myself. I wrote here about the guts of GLM-5.2: open weights, MIT licence, frontier-adjacent performance… and an awkward 376 GB reality that almost nobody can run under their own roof. That post’s conclusion still stands: open is not the same as free. But notice which way the criticism points. GLM-5.2’s problem was never that its weights were public. It was that the infrastructure to use them isn’t. The right answer to that isn’t closing the weights further — it’s opening the infrastructure. The same goes for the mirror-image illusion: a “sovereign” AI that quietly calls somebody else’s API isn’t sovereign — it’s a hermit crab with excellent marketing.

The exam only the incumbents can afford

Here’s my concrete disagreement, and the one section where I get prickly.

Of Amodei’s three measures, two are defensible within the very framework he proposes. The third — mandatory safety testing for every sufficiently capable model — is right in principle and extremely dangerous in implementation. Because an exam is not neutral: it has a cost, and different players pay that cost very differently.

A lab with billions in funding absorbs a mandatory evaluation regime the way it absorbs an electricity bill: it has the policy team, the lawyers, the in-house evaluators and, above all, the relationship with the regulator that lets it help write the exam. A European university lab, a twelve-person startup, or a collective publishing a model on Hugging Face does not. For them, the same requirement isn’t a bill — it’s a wall.

Nobody has to act in bad faith for this to end badly. It’s enough for each actor to reasonably defend their own interests and for the burden to be applied flat. The outcome — market consolidation in the hands of whoever’s already inside — is identical to what a ban would produce, but with better press. Regulatory literature has had a name for this since the seventies: regulatory capture. Its classic symptom is exactly this — a rule whose compliance cost is flat and whose competitive impact is brutally asymmetric.

To be clear, because the nuance matters: I am not saying this is Anthropic’s intent. The post itself insists the tests apply equally to open and closed models, American and foreign, which is the opposite of a privilege for insiders. I’m saying something duller and harder to dodge: good intentions don’t survive the legislative process, and the details of who pays for the exam, who designs it, and above what threshold it applies will decide more about the ecosystem’s future than the entire philosophical argument over whether weights should be public.

What I’d argue for

If they let me write three lines into that law, these would be them.

A threshold based on measured capability, not on an adjective. Let the trigger be what a model demonstrably does on concrete evaluations — biology, cyber-offence, autonomy — not the licence it ships under or the size of whoever published it. Amodei and I agree here, and that’s worth underlining.

Proportional cost and public evaluation. Build a publicly funded evaluation infrastructure — a CERN for evals — where a small team can submit a model without going bankrupt. If safety is a public good, verifying it can’t be a paid service only five companies can afford.

And an honesty test for everyone. Amodei asks that the safety benefits claimed by open-weights advocates be demonstrated empirically rather than assumed. Entirely fair. Apply it in the other direction too: let closed labs demonstrate empirically that their opacity produces more safety than the scrutiny of thousands of outside researchers. Neither claim should travel for free.

What I’m taking home

The real disagreement isn’t “open yes, open no.” It’s where we set the threshold, who measures it, and who pays for the measurement. Everything else is a headline war.

Efficiency and resilience are not the same virtue. The monoculture wins every year until the year it loses everything. An ecosystem with open, small, auditable, run-it-at-home models scores worse on the benchmark and far better in the disaster. And in Europe, where we don’t have a frontier of our own but we do have regulation, that distinction is literally our industrial policy.

Irreversibility cuts both ways, and both deserve to be looked at straight. An open model can’t be pulled when it turns out to be dangerous — that’s Amodei’s argument, and it’s true. An open model also can’t be pulled when it turns out to be inconvenient for whoever’s in charge — or when a letter arrives from Washington — that’s mine, and it’s also true. The question isn’t which sentence is true; both are. It’s which of the two risks frightens you more. Watching compute concentrate into three postal addresses, I’m fairly clear about which one keeps me up at night.

The Svalbard seed vault wasn’t built because someone knew which harvest would fail. It was built because someone accepted that one of them would.

Red pill swallowed, as always.

Sources: