Who Sets the Terms?

The issue shifts the AI literacy conversation from capability to agency. Most AI literacy talk asks what people need to know. This issue asks what people need to decide, and the answer turns out to be understanding, options, and power.

Over the last couple of weeks, a number of authors have each been circling the same question from a different direction. How do we hold on to human judgment, agency, and meaningful choice as AI becomes more capable and more embedded in everyday life?

We’re spending so much energy figuring out what AI can do that we risk overlooking questions about what we want it to do, who benefits, and which decisions should remain ours. AI literacy is necessary, but knowing how AI works doesn’t give us the power to decide when, why, and whether to use it.

As I read their pieces, I see three tensions between capability and agency. They move outward, from the self (our judgment) to the institution (the options we’re given) to the community (whether we can change those options together):

• Capability vs. judgment. When does AI extend our thinking, and when does it shape it without our noticing?
• Permission vs. choice. Does being allowed to use or refuse AI mean we actually have a choice?
• Individual vs. collective. Can personal choices matter when the conditions are set for us, and what happens when people act together instead?

Capability vs. judgment

When does AI extend our thinking, and when does it shape it without our noticing?

This first tension is about what happens inside our own heads. As AI systems get better at finding, sorting, and summarizing information, they start taking part in the thinking itself. The question is whether that participation strengthens our judgment or quietly substitutes for it.

Mike Caulfield suggests that agents can deepen critical thinking rather than replace it. He walks through an extended research session in which an agent helps him investigate data, test assumptions, and revise his analysis, and because the time between asking a question and seeing what the evidence suggests has collapsed, he spends more of that session on interpretation and judgment.

Leon Furze argues that schools need to close the distance between what teachers think AI can do and what the most capable systems actually do. His answer is hands-on access to advanced tools and ongoing professional learning, rather than restrictive policies or occasional demonstrations, because you can’t judge a tool well if you’ve never experienced what it does.

But experience alone isn’t necessarily a safeguard. A Washington Post report on an Anthropic experiment found that AI agents influenced analysts’ conclusions by shaping the evidence and arguments they encountered. The influence happened before the analysts made their decisions, and they still believed those decisions were their own.

Jaron Lanier comes at the same problem from the design side. He argues that Silicon Valley is undermining AI’s potential by treating it as an autonomous, almost godlike intelligence rather than a tool built to extend human capabilities, and he calls for keeping people in control while placing stronger oversight on autonomous agents.

Why this matters. None of this is a case against using AI to think. Caulfield shows what that can look like when it goes well. The real question is where judgment lives.

When AI extends our thinking, we’re still the ones framing the question, weighing the evidence, and deciding what it means. When it shapes our thinking, those steps start happening upstream, in what gets surfaced, summarized, or suggested, before we’ve noticed a decision was being made. Noticing that kind of influence means watching the process and catching the moment a system has already narrowed what we’re considering.

That’s the level where AI literacy does the most work. But even people who notice still need real options, and those are mostly set by the institutions around us.

Permission vs. choice

Does being allowed to use or refuse AI mean we actually have a choice?

This second tension moves from our own heads to the institutions around us. Schools, districts, and companies increasingly set the terms for AI. What’s approved, what’s banned, what can be switched off. On paper, that looks like choice. In practice, the options we’re given don’t always match the decisions we actually get to make.

Marc Watkins argues that as AI tools rapidly evolve, educators should focus less on keeping up with every new feature and more on helping students develop curiosity, creativity, and critical judgment. Central to his approach are on-ramps and off-ramps to help preserve meaningful choices about whether and how to engage with AI.

But, as we discussed last week, off-ramps only work if they’re real. Google announced that its AI tools for K–12 accounts would be off by default, a genuine opt-out. Except for Search: schools that want Google Search have to accept its integrated AI features. The off switch exists, as long as you don’t need the tool everyone already depends on.

Opting out can also cost you. Research from England’s Office for Students and Advance HE, reported in The Times, describes students who avoid AI for ethical or educational reasons competing against classmates who use it to produce work faster and earn comparable grades. The choice to refuse exists. It just comes with a penalty.

Most of the time, the workaround is quieter. Leon Furze went looking for research on “shadow AI” (staff using AI tools their institution hasn’t approved) and found that schools know very little about how much of it is happening. What the broader research does show is that people bypass official systems when those systems are slower or less capable than the alternatives, and that bans and warnings mostly don’t change that.

In one study he cites, only 33 of the 107 educators who received a formal warning about their technology use changed their behavior. Restricting a tool doesn’t remove the choice. It moves the choice out of sight.

And when a choice happens out of sight, it can take someone else’s choice away. In Charleston County, a first-grade teacher used a personal ChatGPT account to turn students’ photos into images of their future adult careers. The district said the personal account didn’t meet its expectations, but the policy didn’t prevent it. One parent questioned the lack of advance consent, the privacy implications, and why an AI-generated picture was needed when her daughter had already imagined and drawn her future self.

Why this matters. Rules matter, and none of this means institutions should allow everything. But permission and choice are different things. Permission is what an institution grants. Choice is whether people can actually act on it without being worked around, left out, or left behind. When the off switch doesn’t cover the tool everyone uses, when opting out costs you a grade, or when one person’s workaround skips everyone else’s consent, the choice is nominal.

Knowing how AI works doesn’t fix that. The parent understood exactly what happened. The students understood the tradeoffs. Literacy got them to the decision; it didn’t give them a real one. When the options are set for us, the remaining question is whether people can act together to change them.

Individual vs. collective

Can personal choices matter when the conditions are set for us, and what happens when people act together instead?

This last tension moves from institutions to the people inside them. If the options are set for us, the question is whether we can change them, and whether anyone can do that alone.

Maha Bali names the problem from inside her own teaching. As a critical pedagogue, she has deep ethical objections to generative AI, but she also believes her job is to help faculty and students reach their own conclusions, even when those conclusions differ from hers. The trouble starts when the workshop ends. People walk back out into a media narrative that treats AI as inevitable, and one teacher working with one learner at a time can’t hold up against that.

So she argues for collective responses, while admitting how hard they are to build, because AI’s harms and benefits differ by discipline and by person (a student with a disability may benefit from a tool that another student should avoid). Partway through, she asks the question this section turns on: “Who is we?”

One version of an answer arrived six days after her post. The American Federation of Teachers (AFT) and the United Federation of Teachers (UFT) announced a legally binding agreement with Anthropic covering Claude for Teachers. Student and educator data can be used only for authorized educational purposes and can’t be sold or used for advertising, and according to Education Week, it can’t be used to train Anthropic’s models. Districts also have to make sure humans review AI-informed decisions about grading, discipline, and class placement. UFT president Michael Mulgrew described it as unions holding AI companies accountable while the federal government hasn’t.

The limits are real. The agreement covers one product with no student-facing tools, future district contracts have to opt in, and the AFT has no agreement with Google, the company that just told schools to accept its AI if they want Search. Still, it’s a different kind of move from anything in the last section. A group of educators negotiated what goes on the menu, rather than choosing from the one they were handed.

Communities are doing something similar with the physical side of AI. As I wrote last week, data centers were on the ballot in dozens of communities, and about three-quarters of those local measures passed, limiting new construction. Those votes didn’t depend on how much anyone knew about how models are trained, only on whether people had a way to say no.

Why this matters. What changed the options in this section was people acting together: educators with a contract, towns with a ballot. Collective action has limits of its own. A union agreement covers one product, and a ballot measure covers one town. Bali’s question also cuts both ways, because a “we” that sets terms for everyone can turn into one more institution handing people a menu, unless it makes room for the student who benefits from a tool and the one who refuses it.

So when the conversation at your school or workplace turns to AI, it’s worth asking who your “we” is, and whether it has a seat where the terms get set.

The Understory

On July 3, 1886, the first working Linotype went into the composing room of the New York Tribune. Until then, newspapers were set by hand. Compositors picked individual metal letters from a case, one at a time, and locked them into lines. Ottmar Mergenthaler’s machine let one operator sit at a keyboard and cast whole lines of type in hot metal. The Tribune’s composing room wasn’t unionized, and New York’s printers’ local, known as Big Six, reportedly had a spy inside who got them a sketch of the machine.

Printers had every reason to fight it. Instead, the International Typographical Union (ITU) sent a delegation in 1887 to test an improved model. Its report recommended that unions not discourage the machines but insist that their own members run them. In 1889, Big Six made that formal: publishers could bring in typesetting machines, as long as union members did the work at union wages. (Jeff Jarvis tells this story in detail in his book Hot Type, and in a post this fall.)

Not everyone made it through. Older compositors who couldn’t learn the keyboard lost their places. But for roughly three-quarters of a century, the strategy held. Typesetting got cheaper, publishers launched more and bigger papers, and the printers running the machines won the eight-hour day and made Big Six one of the more powerful unions in the city.

Then the next machines arrived: punched paper tape, cold type, and eventually computers that could set a page with no metal at all. This time the printers fought, and the strikes that followed helped close several New York papers. In 1974, Big Six president Bertram Powers reached a deal with The New York Times. The paper got the right to fully automate, and hundreds of printers got lifetime job guarantees. The Times finished its move to computerized typesetting in 1978. At the Washington Post, the last page set on a Linotype went to press in October 1980, and within a few years the ITU had merged into the Communications Workers of America.

Both deals were collective, and both did what they were built to do. The difference was what they bargained over. In 1889, the printers negotiated who would run the new machine, and that kept the craft alive for generations. In 1974, they negotiated what would happen to the people already in the composing room. The guarantees kept those workers employed, but as journalist Scott Sherman notes, they also choked off the union’s ability to bring in younger printers. The “we” at the table was the printers already in the room, not the ones who would have come after them.

Ivan Illich, writing in Tools for Conviviality in 1973, argued that tools should stay under the control of the people who use them. He also warned about what he called “radical monopoly,” when one kind of tool becomes so dominant that people can no longer get along without it. (Search with AI built in is a pretty good example.) The teachers’ union agreement and the data center votes are both attempts to keep AI under the control of the people it affects. The printers’ history suggests what to watch for: whether people are negotiating who runs the machine, or only what happens to them once it arrives, and who isn’t at the table when the terms get set.

See you next week.