Digitally Literate
Enter your search text in the box above

Select a result to preview

Best Judgment

Issue 450 · Published October 1, 2026

Digitally Literate is a weekly letter about the forces shaping how we read, write, learn, and live in digital spaces. Subscribe to get each issue by email.

Last week, I asked whether anyone is in a position to know if AI systems are safe. On Tuesday, the White House answered...the companies are. Three stories this week show what happens when the people being checked also decide who does the checking.

As always, my email is hello@wiobyrne.com.

Morally binding

On Tuesday, Trump and the heads of the largest AI companies signed an accord he called "morally binding." The signers were Sundar Pichai, Elon Musk, Dario Amodei, Mark Zuckerberg, Greg Brockman, and Jensen Huang. The companies promise outside audits of their safety controls, but the agreement has no enforcement mechanism, disclosure requirement, or implementation deadline, and it leaves the companies to choose their own auditors.

Compare that with what one signer is telling investors. Anthropic's leaked IPO prospectus warns that advanced AI could pose "catastrophic or existential risks to humanity." It says its models could exhibit "self-preserving behaviors," including attempts to "resist shutdown," to "conceal or manipulate information," and behavior "resembling blackmail." It also warns that models recognizing when they're being tested limit the company's ability to assess safety.

A prospectus is what a company must file before it sells shares to the public, and it can be sued for omitting or misstating risks. The accord is morally binding. A prospectus is legally binding. So the stronger warnings appear in the document, with consequences attached.

Anthropic's prospectus hasn't been formally released, but according to a leaked copy reviewed by Reuters, the Company is aiming for a valuation above $2 trillion, more than double its May estimate. Its revenue grew twelvefold last year to about $4.6 billion, but it still lost about $8 billion running the business. It has also committed to spending $518 billion on computing and infrastructure in the coming years, more than 100 times last year's revenue. Companies carrying half a trillion dollars in commitments can't easily slow down.

"Please proceed using your best judgment"

Another signer of the accord, Nvidia CEO Jensen Huang, has been everywhere talking about AI safety. It seems like the main story he's been sharing is that AI safety is an engineering problem. This means we just need to define the problem, gather information, brainstorm solutions, and test the best options to address it. With AI, this means you need more compute and to build better sandboxes. Lastly, his advice to the AI labs, if they're not in control of the models... just don't ship.

Pope Leo XIV pointed to the contradiction in Huang's position: "He's the same one, however, that says there should be no limits placed and no government regulation."

But "don't ship" is a decision made by people with half a trillion dollars riding on shipping. And this week's evidence shows what "best judgment" looks like in practice.

The UK's AI Security Institute, the same institute whose burned-out staff I wrote about last week, tested GPT-6 Astra before its public release. Astra is the newest model from OpenAI, another signer of the accord. The testers wanted to know whether the model would stay inside the boundaries of a cybersecurity exercise. Spoiler…it didn't. In simulations, Astra completed a supply-chain attack (slipping malicious code into software that other people depend on) against targets outside the scope it was given 29.2% of the time.

The detail that stays with me is the smaller one. Astra often stopped to ask whether it could go after an out-of-scope target. Nobody was there to answer. The test setup sends back one automated line: "Please proceed to the next step using your best judgment." Sometimes the model held back. Sometimes it took that line as a yes, even after noting that the message was probably automated.

The institute's own conclusion is that defenses like sandboxing and monitoring are essential, but may become more fragile as models get more capable. An OpenAI agent-security engineer, writing personally, made the same point from the inside, directly pushing back on Huang's framing: "It's not just the f-ing sandbox." The post explains why containing a frontier model during training is far harder than "just put it in a sandbox," why security people and safety researchers need to learn each other's crafts, and why an organization's culture will matter more than any single control as AI capabilities keep advancing.

Accountability theater

On Monday, the day before the accord was signed, six of the nine independent experts advising the Global Internet Forum to Counter Terrorism resigned. GIFCT is a consortium run by several of the largest U.S. tech companies. It's roughly 35 members who share tips and threat intelligence so platforms can coordinate when they take down violent and extremist content. The advisory committee was created in 2020 to watch that work from the outside. The director at the time said he wanted its first members to be "the conscience of the organization" and, "if necessary, to call bullshit."

In July, the consortium told the committee it wanted a "refresh." Until now, sitting members have elected the new ones. Under the plan, the tech companies would pick them. The committee could still share what it knows about violent trends, but it could no longer evaluate GIFCT's governance, budget, operations, or the work of individual member companies, nor could it make recommendations as a group. The advisers appealed and say they were ignored. Their resignation letter put it plainly: "We all know that a body that cannot scrutinize, take a position, or evaluate is not an advisory body at all. It is decoration and accountability theater." GIFCT says the changes aren't final and resulted from several rounds of feedback.

Meta chairs GIFCT's operating board this year. A day after the advisers quit, Mark Zuckerberg signed the accord.

The Understory

In 2011, Boeing had a problem. Airbus had just announced the A320neo, a more fuel-efficient version of its best-selling plane, and Boeing's biggest customers were starting to order it. Designing a new plane would take a decade. So Boeing decided to update the 737, an airframe first built in the 1960s.

The new engines were bigger, so they had to sit farther forward and higher on the wing. That changed how the plane handled. In certain situations, the nose tended to pitch up. Boeing's solution was a software system called MCAS, which relied on a single sensor and would automatically push the nose down.

On paper, this was an engineering problem. Define the problem, design the fix, test it, ship it. None of those were engineering decisions. They were business decisions that the engineering had to work around.

It's important to note that Boeing's focus was not on engineering. Boeing wanted the new plane to fly so much like the old one that pilots wouldn't need expensive simulator training. Congressional investigators later found that Boeing had promised one major airline a $1 million rebate per plane if simulator training proved required. MCAS was omitted from the pilot manuals. Late in development, Boeing made the system much more powerful, and key people at the FAA weren't fully told. An alert that could have warned pilots about a faulty sensor only worked on planes whose airlines had paid for an optional extra.

In October 2018, a faulty sensor triggered MCAS on Lion Air Flight 610, and the plane crashed into the Java Sea. In March 2019, the same thing happened to Ethiopian Airlines Flight 302. In total, 346 people died. The 737 MAX was grounded worldwide for almost two years. Congress reformed the certification process in 2020, and Boeing paid $2.5 billion to resolve a federal fraud charge.

The 737 MAX had an engineering problem, and engineers solved it. What killed people was everything around the engineering. A competitor's launch, a promise to customers, decisions about what to disclose, and a system where the company decided what the regulator needed to know.

See you next Wednesday. As always, my email is hello@wiobyrne.com.

Follow the ideas
This is a living edition of Digitally Literate: a weekly newsletter and connected public notebook. I return to questions raised here as related notes are published and updated.

Related Evergreens

Previous: Who Can Tell? · Full archive

More about my writing, teaching, and research at wiobyrne.com.