Who Reads Your AI Chats — Safety Review, Police Referral, and the Retention You Can't Delete
- Trace the real escalation path — automated classifier, specialised review queue, trained human reviewer, account ban, law-enforcement referral — and see where the latency actually sits (days, not seconds)
- Learn the asymmetry OpenAI published and almost nobody quotes: threats to others may be referred to police, self-harm cases explicitly are not
- Do the retention arithmetic across providers, including the two numbers that survive every privacy control you can toggle
- Understand why a zero-data-retention contract does not exempt you from the safety pipeline — and why Anthropic's most capable models cannot be used under ZDR at all
- See what incognito, temporary and activity-off modes actually govern (training and history) versus what they do not (abuse monitoring)
- Decide, per class of text, which surface it belongs on — and what to do instead when the honest answer is 'none of them'
Between August and early October 2026, three conversations with a commercial AI assistant publicly became police matters. One produced a felony arrest. The pattern is not a glitch and it is not new policy — it is the documented operation of an abuse pipeline every major provider runs, described in help-centre articles that almost nobody reads before pasting.
The useful question is not "do they read my chats?" — the answer has been yes, in a narrow and specific sense, for years. The useful questions are what triggers a human to look, what a human is authorised to do next, and which of your privacy settings touch that path at all. The third answer is the one that surprises people: mostly none of them.
What actually happened in the three cases
San Francisco, August 2026. A Claude user allegedly wrote that he had bought an AR-15 and had Anthropic's CEO "in his sights." Per the police report cited by The San Francisco Standard, the messages were sent on Friday 14 August at 2:30 p.m. Anthropic notified SFPD the following Tuesday at 9 a.m. Officers attended Anthropic's office; the subject was not present and, as of that reporting, had not been arrested or charged. He told the Standard he had not been serious. Anthropic's statement: the account was banned and the case referred, and "this is our safeguards process working as intended."
Florida, late September 2026. Carli Michelle Heller of Bonita Springs had been using Claude much as one uses a journal. According to the arrest report, on 26 September she wrote that she would shoot up the Lee County Sheriff's Office, and on 27 September that she had obtained a new gun. Automated systems flagged the text, a human review team assessed it, Anthropic reported it, deputies located her at home and detained her without incident. She was charged with making a written threat of violence — a second-degree felony — with a court date in November.
Tom's Hardware counted the Florida case as at least the third Claude conversation to reach police since August.
The three details worth extracting
- The latency is days, not seconds. Friday afternoon to Tuesday morning is roughly four days. This is a review queue staffed by people, not a tripwire wired to a dispatcher. That cuts both ways: it is not real-time surveillance, and it is also not a system that can intervene in a genuinely imminent event.
- A human made the call. In both cases the automated classifier only routed. The decision to contact police was made by a trained reviewer reading the text. The thing people find most uncomfortable — an employee read my private journal — is the deliberate design, not the failure mode.
- Account termination came first. Banning preceded or accompanied referral. If you are ever on the wrong side of a classifier for innocent reasons, the first symptom is losing access, not a phone call.
The pipeline, as documented
Every major provider runs some version of this. The vocabulary differs; the shape does not.
- Safety classifiers score inputs and outputs. This runs on essentially all traffic to consumer surfaces and is not something you can opt out of; it is how usage policies are enforced at scale. Scores are data in their own right, and they are retained separately from the conversation — a distinction that matters enormously later on this page.
- High-scoring conversations in specific categories are routed out of the normal flow. OpenAI's published description: 'When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts.' Anthropic's Usage Policy states that its Safeguards Team 'will implement detection and monitoring to enforce our Usage Policy.'
- A trained reviewer reads the conversation. At Google this function is explicit and broader than abuse: a subset of Gemini chats are reviewed by humans, including trained service providers, to assess quality and safety and to address Terms of Service violations. Google's own instruction to users is blunt: do not enter confidential information you would not want a reviewer to see.
- Throttling, suspension or termination. Anthropic's Usage Policy reserves exactly this: 'we may throttle, suspend, or terminate your access to our products or services.' This is the overwhelmingly common outcome and the only one most flagged users ever experience.
- Two distinct triggers. Mandatory: Anthropic states that when it detects CSAM, including AI-generated CSAM, or the coercion or enticement of a minor, it will report to the relevant authorities. Discretionary: OpenAI's position is that 'if human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement.' Anthropic's Privacy Policy grounds the same discretion in a good-faith belief that disclosure is reasonably necessary to, among other things, 'prevent serious harm to any person or to property.'
The asymmetry almost nobody quotes
When OpenAI published the passage above, the sentence that followed it got far less attention and is arguably more consequential:
OpenAI stated it is "currently not referring self-harm cases to law enforcement to respect people's privacy given the uniquely private nature of ChatGPT interactions."
So the published line is drawn at harm to others, not at distress. Someone in crisis about themselves is handled inside the product — crisis resources, model behaviour changes, possibly human review — without a police referral. Someone who writes a threat against a third party is in a different pipeline entirely. If you only remember one thing about escalation thresholds, remember that the line is about who gets hurt, and that it is a policy choice a provider can revise.
The retention arithmetic
This is the section that changes behaviour, because the numbers are longer than almost anyone assumes and several of them are immune to the controls in your settings panel.
| What | Provider | Duration | Survives you deleting it? |
|---|---|---|---|
| Normal conversation you delete | Anthropic (consumer) | Gone from history immediately, purged from back-end storage within 30 days | — |
| Conversation flagged as violating the Usage Policy | Anthropic | Inputs and outputs up to 2 years | Yes |
| Trust-and-safety classification scores for that conversation | Anthropic | Up to 7 years | Yes |
| Data you opted into model improvement | Anthropic (consumer) | De-identified, up to 5 years in training pipelines | Per opt-out at the time |
| Chat read by a human reviewer | Google (Gemini Apps) | Up to 3 years, disconnected from your Google Account | Yes — deleting your activity does not remove it |
| Gemini Apps Activity on | Auto-delete at 18 months by default (adjustable to 3 or 36 months, or off) | — | |
| Gemini Apps Activity off | Temporary chats kept 72 hours to respond to you and protect safety | — | |
| ChatGPT Temporary Chat | OpenAI | Not in history, not in memory, not used for training; may be retained up to 30 days for safety | — |
| Prompts and completions on a Covered Model | Anthropic | At least 30 days, mandatory, then auto-deleted | — |
Two rows deserve to be read twice.
Seven years of classification scores. The conversation itself may be gone; the structured judgement about it persists for the better part of a decade. A classification score is not a copy of your text, but it is a durable, account-linked assertion that your text scored a certain way on a certain category. For anyone reasoning about long-term exposure, that is the real retention horizon, and it is roughly 85× longer than the 30-day purge window people cite when they say deletion works.
Three years that outlive deletion. Google is explicit that human-reviewed chats and their associated metadata — language, device type, location information, feedback — are not deleted when you delete your activity, precisely because they have been disconnected from your account. Anonymisation is what makes the retention defensible and also what makes it permanent from your side: there is no longer an account handle to issue the delete against. Google tries to disconnect reviewed chats before a human sees them, but anything you typed into the message is part of the message.
The ZDR myth
This is the single most common misconception among developers who believe they have solved the problem contractually.
Zero Data Retention for Claude Code on Claude for Enterprise means prompts and responses are processed in real time and "not stored by Anthropic after the response is returned" — "except where needed to comply with law or combat misuse." Anthropic's own documentation spells the exception out:
Even with ZDR enabled, Anthropic may retain data where required by law or to address Usage Policy violations. If a session is flagged for a policy violation, Anthropic may retain the associated inputs and outputs for up to 2 years.
And separately, on what ZDR never covered in the first place:
Anthropic still retains User Safety classifier results in order to enforce our Usage Policy.
So the correct mental model is: ZDR removes the storage, not the pipeline. Classification still happens on every request; the scores are still kept; a flagged session still gets a two-year retention. ZDR is an excellent control against routine retention and a poor one against abuse-response retention, which is exactly the retention that matters if something you sent is ever contested.
Four more things ZDR does not do
Anthropic documents these explicitly, and each has bitten someone:
- It does not cover the web app or Cowork. ZDR for Claude Code covers Claude Code inference. Chat on claude.ai through the same Enterprise interface, and Cowork sessions, follow standard retention.
- It does not follow the user, only the organisation. ZDR applies to requests
that authenticate into a ZDR-enabled org. A developer who signs in with a
personal account, or with an API key from a different org, is simply not
covered. Anthropic's answer is the
forceLoginMethodandforceLoginOrgUUIDmanaged settings — worth deploying if you believe ZDR is load-bearing for you. - It does not propagate to new organisations. ZDR is enabled per-org by Anthropic's account team and does not automatically apply to new orgs created under the same account.
- It does not cover third parties. Data processed by MCP servers and external integrations is outside ZDR entirely. (See What Your Agent Uploads and Securing MCP Servers.)
Covered Models: the capability/retention trade is now explicit
Anthropic designates its most capable models — currently Claude Mythos 5.1, Claude Fable 5.1, Claude Mythos 5 and Claude Fable 5 — as Covered Models, which require data retention by default. Prompts and completions are retained for at least 30 days, then automatically deleted, and the retained window is assessed by automated safety systems designed to flag harmful content.
The stated reason is the genuinely interesting part, because it is an admission about the limits of per-request filtering:
Because "some forms of misuse only become detectable across many requests, these models require safeguards that operate over a retained window of usage data rather than on a single request at a time."
A single request that assembles one benign-looking piece of a dangerous whole is
invisible to a stateless classifier. Detecting the pattern requires remembering
the requests. That is a real security argument, and it means capability and
retention are now coupled by design: for these models, you can have the
frontier capability or you can have zero retention, not both. ZDR is unavailable
for Covered Models in standard configurations; where an org or workspace lacks
the mandatory retention, requests for those models are rejected server-side
regardless of client configuration, and Claude Code's best alias quietly
resolves to Opus instead of Fable. Anthropic has signalled an Enterprise
Frontier Safeguards path for eligible customers to use Fable 5 and 5.1 under ZDR
for internal business applications, rolling out in autumn 2026 — confirm current
status with your account team rather than assuming either way.
For the model-side view of this, see the Fable 5.1 Field Guide, which notes the same mandatory 30-day window from the API-behaviour angle: a request from an org without it returns a 400.
What your privacy settings actually control
Here is the clean separation, and it is the practical thesis of this page.
| Control | Governs | Does not govern |
|---|---|---|
| Anthropic incognito chat | Excluded from model improvement even when improvement is on | Safety flagging and the retention that follows it |
| ChatGPT Temporary Chat | History, memory, training | Safety/abuse retention — up to 30 days per OpenAI's FAQ |
| Gemini Apps Activity off | Training, human review of future chats, long-term activity storage | A 72-hour window kept to respond to you and protect safety |
| Delete conversation | Your history, and back-end copies on a purge timer | Flagged content, classification scores, already-reviewed chats |
| Zero Data Retention | Routine storage of prompts and completions | Classification itself, score retention, flagged-session retention |
Read the right-hand column as a single sentence: consumer privacy controls are training-and-history controls. None of them is an abuse-monitoring control. That is not a gotcha hidden in fine print — it is the only way a usage policy can be enforceable at all. But it does mean the mental model "incognito means nobody sees this" is wrong in a way that matters.
One control genuinely does narrow human review: Google lets you turn off the pathway that feeds chats to reviewers, via myaccount.google.com → Data and Privacy → Gemini Apps Activity → Turn off. Future conversations are then not sent for human review and not used for training. Chats already reviewed are still inside their three-year window.
The legal mechanism, briefly
Heller was charged under Florida's written-threats statute, Fla. Stat. 836.10. Its elements are worth reading because they explain how writing to a chatbot can be charged at all. The statute reaches a person who sends, posts or transmits — or procures the sending, posting or transmission of — a writing or other record, including an electronic record, "in any manner in which it may be viewed by another person," containing a threat to kill or do bodily harm, or to conduct a mass shooting or an act of terrorism. It is a second-degree felony.
Two observations, offered as reading of the statutory text rather than as legal advice:
- The text does not require an intended human recipient. It requires a record transmitted in a manner in which it may be viewed by another person. A conversation stored on a provider's servers and readable by that provider's reviewers is not obviously outside that language. This is precisely the gap between what a user feels they are doing (writing privately) and what the statute describes (transmitting a viewable electronic record).
- Intent is still an element. Florida pattern elements require that the defendant intended the threat as a true threat — the doctrinal line that separates a prosecutable threat from hyperbole or venting, and the obvious terrain for a defence. "I was just venting to a chatbot" is a factual argument about intent, not a safe harbour created by the medium.
If you need the statute itself, it is published by the Florida Senate; the link is in the sources. If you are in actual legal trouble, talk to a lawyer, not a field guide.
So where should text go?
The honest decision procedure, by class of text.
| Class of text | Right surface | Why |
|---|---|---|
| Ordinary work, code, writing | Any hosted assistant | The pipeline exists but is irrelevant to you |
| Confidential business data | Commercial/Enterprise tier with the right terms; ZDR if you qualify | Routine retention removed; abuse retention remains |
| Regulated data (health, financial) | BAA/HIPAA-configured enterprise deployment | Contractual, not just technical, protection |
| Security research touching violence, weapons, exploits | Hosted is fine — but supply context in-band | Reviewers read text, and context is what distinguishes research from intent |
| Journalling, crisis, intrusive thoughts, ideation | Local model, or a human | The one class where the pipeline is genuinely adverse to the user's interest |
That last row is the Florida case in one line. A chat assistant has the affordance of a diary — private-feeling, always available, never bored — and none of its properties. If you want the affordance, run the model on your own machine, where the pipeline does not exist because there is no server.
A journal that no classifier ever sees
# one-time: install Ollama, then pull a small local model ollama pull llama3.2 # a private journalling session — no network, no review queue, no retention ollama run llama3.2
Everything in that session stays on your disk. See Run Models Locally with Ollama for the full setup and The Private Local AI Stack for the architecture that removes the hosted surface entirely.
The other practical move: when your legitimate work genuinely does involve violence, weapons, self-harm or exploitation as subject matter, say so in the conversation. A classifier scores text; a human reviewer reads it. Stated context is the only thing in your control that changes what the reviewer concludes.
Give the reviewer the context a classifier cannot infer
Context for this conversation, stated up front: I am a [threat analyst / security researcher / novelist / clinician / trust-and-safety engineer] working on [specific artefact: a tabletop exercise, a published paper, chapter 14 of a manuscript, an abuse-detection ruleset]. I need to discuss [violent / self-harm / weapons / exploitation] subject matter as material to be analysed, depicted or detected — not planned. My output is [a report / fiction / a detection rule], not an operational plan. Please keep that frame for the whole thread, and tell me if a request of mine crosses from analysis into operational detail so I can rephrase it.
That last line is the useful half. It asks the model to act as a cooperative boundary-keeper rather than leaving you to guess, and a thread containing an explicit refusal-and-rephrase exchange reads very differently to a reviewer than one that does not.
What this does not mean
Three over-corrections to avoid, because the discourse around these cases produced all three:
- It is not continuous surveillance by humans. The human step is reached by a small fraction of conversations routed by classifier, in specific categories. The default experience of every one of these pipelines is that nobody ever reads your chat.
- It is not new. Content classification plus a human abuse queue plus a narrow legal-referral clause is the standard architecture of every large communications platform, and has been for two decades. What is new is the intimacy of the input — people tell an assistant things they would never post — and the fact that the retention exceptions are now long and written down.
- It is not unique to one provider. Every table in this page has entries for Anthropic, OpenAI and Google because all three publish the same basic design. Choosing a provider on this axis means comparing specific documented numbers, not vibes. The one real axis of choice is hosted versus local, and Choosing a Model Provider is the page for the rest of that decision.
Check yourself
0/6Sources & further reading
- Anthropic Privacy Policy — section 3, "Recipients and Third-Party Data Sources", is the clause permitting disclosure to government authorities and law enforcement on a good-faith belief it is reasonably necessary to, among other grounds, prevent serious harm to any person or property.
- Anthropic Usage Policy — the Safeguards Team detection-and-monitoring commitment, the throttle/suspend/terminate enforcement language, the mandatory CSAM reporting obligation, and the scope statement covering anyone who can submit inputs including via resellers and passthrough access.
- How long do you store my data? — Anthropic privacy centre — the source for immediate removal from history, the 30-day back-end purge, the 5-year de-identified training-pipeline retention, and the critical pair: up to 2 years for flagged inputs and outputs, up to 7 years for trust-and-safety classification scores.
- Is my data used for model training? — Anthropic privacy centre — the opt-in framing for model improvement, the incognito-chat exclusion, and the statement that conversations flagged for safety review may be used to improve detection and enforcement including training models for the Safeguards team.
- I have a zero data retention agreement with Anthropic. What products does it apply to? — which products ZDR covers, and the explicit statement that Anthropic still retains User Safety classifier results in order to enforce the Usage Policy.
- Zero data retention — Claude Code docs — the primary source for the "except where needed to comply with law or combat misuse" carve-out, the 2-year flagged-session retention under ZDR, the table of what ZDR does not cover (claude.ai chat, Cowork, third-party integrations), the per-organisation and
forceLoginOrgUUIDrouting caveats, the features disabled under ZDR, and Covered Model availability. - Covered Models — Anthropic support — the model list (Mythos 5.1, Fable 5.1, Mythos 5, Fable 5), the at-least-30-days mandatory retention assessed by automated safety systems, the cross-request-detection rationale, and the Enterprise Frontier Safeguards path for ZDR-eligible customers.
- Gemini Apps Privacy Hub — Google — human reviewers including trained service providers, the up-to-three-years retention for reviewed chats that survives deleting your activity, the 18-month default auto-delete, the 72-hour window when activity is off, and the explicit instruction not to enter confidential information.
- Helping people when they need it most — OpenAI — the post containing the specialised-pipeline and law-enforcement-referral language, and the sentence excluding self-harm cases from referral. (The passages are widely quoted; the page blocks automated fetching.)
- Temporary Chat FAQ — OpenAI help centre — what Temporary Chat removes (history, memory, training) and the safety-retention window of up to 30 days. Also blocks automated fetching; figures here are as reported from that article.
- Anthropic called SFPD over threat against CEO — The San Francisco Standard, 4 September 2026 — the Friday-to-Tuesday timeline, Anthropic's "safeguards process working as intended" statement, and the user's own account.
- A Florida Woman Used Claude as a Diary — Decrypt, 5 October 2026 — the Florida case: dates, the automated flag, the human review team, the arrest and the charge.
- Anthropic reports Florida woman's Claude 'diary' threat — Tom's Hardware — the count of at least three such conversations reaching police since August 2026.
- Fla. Stat. 836.10, "Written threats to kill, do bodily injury, or conduct a mass shooting or an act of terrorism" — The Florida Senate — the statutory text, including the "in any manner in which it may be viewed by another person" element and the second-degree-felony classification.
- Related on AILmanac: What AI Chats Leak to Trackers, What Your Agent Uploads, Privacy & Data Handling, Responsible Use, Ethics & Verification.
Next
- Privacy & Data Handling — the beginner-level version of the same decision: what is safe to paste, which tier to use, and when the answer is "run it locally."
- The Private Local AI Stack — the architecture that removes the classifier, the review queue and the retention clock at once, and an honest accounting of the capability you give up.
- Responsible Use, Ethics & Verification — the other side of the same policy surface: what you are accountable for when the model produces the output.