Once Your Information Is in an AI Model, You Can't Get It Out

Search engines could honor deletion requests because the data sat at an address in a database. AI models dissolve it into weights — and what is dissolved cannot be extracted.

Ask a simple question — how do I get information about myself removed from an AI model? — and the answer reveals something uncomfortable about how this technology actually works. Posting something untrue about a person takes seconds. Getting it back out of a trained model is close to impossible, and the reasons why say a lot about where privacy is heading.

First, what are weights?

A large language model starts as a blank mathematical slate and learns by reading enormous amounts of text — web pages, books, forums — and adjusting billions of internal numbers called weights. Every time the training process guesses the next word in a sentence and gets it wrong, it nudges those numbers slightly. After months of this across trillions of words, the weights encode the patterns of language and the facts contained in it.

The important part is how they encode it. There is no file labeled with your name, no row in a table, no entry that says this person lives in this city. A fact about you is distributed across millions of weights, blended together with everything else the model read, the way ingredients are blended into a cake. You can taste the egg. You cannot take the egg back out.

When a model answers a question about you, it is not looking anything up. It is recombining patterns baked into those numbers during training. That is the whole reason deletion is hard: there is nothing to point at.

Search engines: the problem that was actually solvable

Contrast this with how Google works, because for twenty years this was the same question with a much happier answer. A search engine operates in three stages: crawling, indexing, and serving. Crawlers download pages from across the web. Everything they find gets analyzed and stored in the index — a vast but ordinary database. When someone searches, Google looks up matching entries in that database and returns links.

That architecture makes removal straightforward, at least in principle. Every piece of information lives at a known address — a URL — as a discrete entry in a centralized index. Delete the entry and the information is gone from the results. Google has built entire products around this. The "Results about you" tool lets people request the de-indexing of contact details, ID numbers, and bank accounts. In Europe, the right to be forgotten — established by a 2014 court ruling — gives people a formal right to ask for delisting; more than 12,000 requests arrived on the first day the form went live.

It is worth being precise about what these tools do. Google removes the result from its own index, not the page from the web. The underlying site still has the information. But the operation is well-defined: one company, one database, one addressable entry. Deletion is an administrative act — reviewable, reversible, and complete once done.

Deletion is a downstream problem

Now run the same request against an AI model and every one of those advantages disappears. The major providers do have request channels. OpenAI lets you submit a request to remove your personal data from model outputs, and their own criteria note that clearly incorrect or outdated information weighs in favor of removal. In Europe, the right to erasure under GDPR gives these requests legal force. In the United States, they are mostly policy goodwill.

But notice what these processes actually do. They stop the model from surfacing the information. They do not remove it from the model. Research on machine unlearning keeps finding the same gap — one study showed that roughly 88 percent of "unlearned" facts could be recovered from a model with a modest fine-tuning attack. The model forgets in conversation. It does not forget in the weights. The real forgetting happens at the next full retrain, months later, on a fresh crawl of the web.

The legal layer is just as unsettled. There is active debate over whether model weights even count as personal data under GDPR. Hamburg's data protection authority has argued they do not — weights are statistical parameters, not a database of facts. Others disagree. That ambiguity is exactly why the industry has settled on a pragmatic answer: do not fix the weights, filter the outputs.

And the false-information scenario is real. If an untrue claim is the only thing published about you when the crawler comes through, that claim becomes part of the model's understanding of the world. It is not necessarily permanent — new generations ship every six to twelve months, and most assistants now ground their answers in live web search, which lets current sources outweigh stale training data. But the window exists, and during that window the false version is the authoritative one for millions of queries.

The local-first counterargument

A post making the rounds this week proposed the obvious escalation. An AI researcher described building a local-first personal assistant: a small 4-billion-parameter model running entirely on the user's device, holding the OAuth tokens for your email and calendar, driving a browser agent on-device, with a built-in skill for filing data broker takedown requests. No cloud COGS, no vendor holding your context, no training on your data. The post leans on the recent Granola lawsuit as evidence that the cloud-first approach carries real legal and reputational exposure.

The thesis is correct, and it is the strongest possible answer to the deletion problem. Unlearning is unreliable, so the only clean delete is the one that never happens. If the data never reaches the vendor, there is nothing to unlearn, nothing to subpoena, and nothing to breach. Everything the assistant learns about you stays in a place you physically control.

The catch

The hard part is the part the pitch glosses over. Browser use, tool calls, and long-horizon tasks are precisely where small models fall apart. "Capable and 100 percent local" is the entire trick, and the history of this industry says you usually get one or the other. Claiming a 2.5-gigabyte model does what frontier assistants with real traction already do is a big swing.

The threat model is also muddled. Local does not mean safe — it means the risk moves. A product that holds your Gmail and Outlook tokens on a laptop and drives a browser agent has concentrated everything an attacker wants on the least hardened device most people own. An endpoint compromise gets everything at once. Most consumer laptops are not more secure than cloud infrastructure; they are just less visible when something goes wrong.

And the marketing deserves its own skepticism. The hook is fear — a claim that anyone can pull a real Social Security number out of a mainstream assistant, unverifiable as stated, and exactly the kind of thing the removal-request process exists for if true. The "audited by top firms" list mixes a private individual, a cloud provider, and a venture firm's internal approval. That is name-dropping, not an audit report.

The bet worth watching

Still, the trend line is real. Small models improve every quarter, and the desire to keep an assistant's full view of your life off someone else's servers is not paranoia — it is a reasonable preference that the current market does not serve. The right way to read this post is as an early privacy wedge for sensitive users, not as a replacement for a frontier assistant today.

The delete button still does not reach the weights. Until unlearning becomes real or the law decides weights are personal data, the only reliable privacy is the kind that never leaves your machine — assuming the machine is finally up to the work.

No comments yet