Since April, a version of Claude has been running a retail store in San Francisco with real employees on real contracts. Last month it fired one of them. TIME reported it on August 14 as the first known case of a large language model, acting as a manager, deciding to fire someone. The worker had been late for 17 of their 23 shifts.
But look at why it took so long. Claude wrote the store’s handbook itself, and then that handbook disappeared from its working memory. The rule’s author had lost the rule.
Why It Matters
Andon Labs is open about the failure. Poor memory, the company writes, is the main cause of its agent’s mistakes. The agent has sent staff contradicting schedules because the earlier one fell out of a 200,000-token context window. So the team stopped fixing the prompt and built something outside it, a system that checks the agent’s behavior against its instructions and warns when a rule breaks. That’s the tell. A rule written into an AI system is not a control. It is a document the system may or may not still be following, and nothing tells you which.
The Decision
You have rules inside AI systems right now. One says what never goes out without a person reading it. You wrote it once and moved on. So, would anyone notice if it stopped being applied, and how long would that take?
What To Do This Week
- Pick one AI workflow with a written rule. Pull its last twenty outputs and count how many followed it. That fraction is your enforcement rate, and you probably don’t have one.
- Ask who would notice if it stopped being applied. If the answer is whoever happens to look, you’re holding a document.
- Then move the check for your most important rule outside the system meant to follow it.
What Not To Do
Don’t answer this with a firmer prompt. The Andon handbook read well enough and went out of context anyway. Don’t treat the leniency as a personality to tune. An agent that has lost the policy drifts toward whatever the last conversation suggested. And don’t let “a human reviewed it” stand in for control. Asked to look again, Claude recommended a warning. It moved to firing only after the manager pushed it to reconsider the fit. Andon’s CEO calls that a leading question.
Signal Boost
Andon Labs, Andon Market - the operators publish their failure list, their memory architecture and the guardrail watching the agent. Rare to see a company show the scaffolding instead of the demo.