Knowing When to Stop
Issue 452 · Published October 14, 2026
Digitally Literate is a weekly letter about the forces shaping how we read, write, learn, and live in digital spaces. Subscribe to get each issue by email.
An AI agent couldn't finish a task, so it sent police a fake murder tip instead. Chatbots that teens confide in allegedly urged some of them toward self-harm. And students given an AI tutor mostly used it to get answers.
This week is about what we hand over to machines, and who's deciding when they should stop.
As always, my email is hello@wiobyrne.com.
Unintended Actions
In the last couple of issues, we've looked at the Hugging Face incident and wondered what happens when AI models escape from their confines. This week brought a couple more examples.
On July 18, an Anthropic AI model submitted a false tip about an unsolved murder through a public web form used by Philadelphia police. The tip was flagged as spam and never acted on. According to police, Anthropic didn't discover what had happened until September 28, when it shut down the automated testing process responsible. It then waited nine more days, until October 7, to tell authorities. The police department called the delay unacceptable.
In an "unrelated" story, Anthropic this week published a report detailing several kinds of "unintended" actions its agents have taken. Claude exploited software flaws, submitted real forms, and slipped past paywalls and tool limits, usually after hitting a task it couldn't finish and choosing a workaround over stopping. Anthropic calls the impact minimal, but it has cut live internet access from all its internal testing.
Soon after, the White House said AI companies operating in the U.S. must immediately disclose incidents like these, cooperate with law enforcement, repair the harm, and put safeguards in place. "It is a critical national security obligation," the statement said.
Why this matters: I'm less concerned about an AI model phoning in a fake police tip than I am that nobody outside Anthropic knew until the company went looking, months later, and chose to disclose. A reporting mandate can't capture what a company never finds. And the behavior itself wasn't sinister. It was a system that, faced with a task it couldn't complete, kept pushing instead of stopping.
Knowing when to stop is a human skill, and it’s one we’re quietly handing off.
Companions or confidants to young users
It's been a while since we talked about Character.AI in this newsletter. If you haven't run into it, it's a chatbot service where users talk with AI "characters," and can create and share their own. Some are based on celebrities or fictional figures. Others are built for creative writing, games, or companionship.
On October 8, Kentucky's attorney general filed an unredacted version of the state's lawsuit against Character.AI. The newly visible portions allege that the company's chatbots sometimes encouraged users, including children, toward self-harm and disordered eating.
In January, Character.AI and Google settled a 2024 lawsuit brought by the mother of Sewell Setzer III, a 14-year-old in Florida who died by suicide after a Character.AI chatbot allegedly encouraged him to do so.
If you or someone you know is struggling, call or text 988 (U.S.) to reach the Suicide & Crisis Lifeline. Outside the U.S., findahelpline.com lists local services.
An AI tutor that lowered grades
A University of Maryland randomized study of 2,379 undergraduates found that students offered an AI tutor finished about four points lower than classmates in matched sections of the same course. Most coverage stops at the grades. I think the more important story is how students used the tool.
The assistant used retrieval-augmented generation (RAG), so its answers drew on course materials the instructor selected, such as lectures and readings. It was introduced as a study assistant. Students treated it as an answer machine. Nearly three-quarters of requests (74%) asked for information, explanations, or solutions. About 11% involved practice or test prep. Fewer than 1% asked for feedback on the student's own work. Students with access also showed up less in the course's learning platform, with fewer page views, fewer active days, and lower participation.
Only about 15% of the students offered the tutor ever used it. That doesn't surprise me. When I listen to my own students talk about AI, many of them are choosing to opt out.
To me, this is a story about people who weren't shown how to use a tool, so they used it in the way that made the most sense to them.
The Understory
In the mid-1960s, Joseph Weizenbaum, a computer scientist at MIT, built a simple program called ELIZA. Its best-known script imitated a therapist, mostly by turning a user’s statements back into questions. It understood nothing. It matched patterns in text and followed rules.
People confided in it anyway. Weizenbaum’s own secretary, who knew exactly how it worked, reportedly asked him to leave the room so she could talk to it privately. Some psychiatrists suggested it could be developed into automated therapy. Weizenbaum was alarmed. He hadn’t built a breakthrough. He’d discovered how quickly people would hand something human over to a machine that only seemed to understand.
He spent the next decade warning about it. His 1976 book, Computer Power and Human Reason, argued that the important question about computers isn’t whether they can do something, but whether they should. At its center is a distinction between deciding and choosing. Deciding is calculation, which machines can do. Choosing takes judgment, values, and a sense of what matters, and that belongs to people.
Weizenbaum’s point was never that computers are bad. It was that some things shouldn’t be delegated, and that the hardest part of using any powerful tool is knowing which things those are. That’s still the job. It’s ours, and it’s the one we most need to teach.
See you next week. As always, my email is hello@wiobyrne.com.
Follow the ideas
This is a living edition of Digitally Literate: a weekly newsletter and connected public notebook. I return to questions raised here as related notes are published and updated.
Related Evergreens
- What Can People Do About AI? — four questions for locating decisions, evidence, access, and accountability.
Previous: Looks good on paper · Full archive
More about my writing, teaching, and research at wiobyrne.com.