We're Testing AI for the Wrong Failure
Demis Hassabis recently published one of the most thoughtful essays I’ve read on artificial intelligence. He argues, correctly, that AI is not another software cycle but a civilizational technology—more akin to electricity or fire—and proposes a new international standards body to evaluate frontier models before they become widely deployed.
I agree with almost all of it. Civilizational technologies deserve institutions that match their scale. My concern is that we are evaluating only one side of the problem.
Today’s AI safety debate is understandably focused on the behavior of the models themselves. Can they help build bioweapons? Can they deceive users? Can they evade human control? These are serious risks, and they deserve serious institutions.
But a system can pass every one of those tests and still leave the people who use it worse off.
A recent randomized study from researchers at Wharton illustrates the point. Nearly one thousand high school students were given an AI math tutor. Half received a version that readily supplied answers. The other half received a version that offered hints while requiring students to continue working through the problem themselves. While using the tutor, both groups improved. Then the tutor was removed and the students were tested independently. Those who had relied on the answer-generating tutor scored seventeen percent worse than students who had never used AI at all. Students who used the guided tutor showed no such decline.
Run that tutor through today’s frontier safety evaluations. It doesn’t help build a weapon. It doesn’t deceive its users. It doesn’t seek to escape human oversight. It is exactly the sort of capable, helpful assistant we are trying to build.
Yet one version quietly left the people who depended on it less capable of solving problems on their own.
One might reasonably object that this is a deployment problem rather than a model problem. I think that’s exactly right.
The model was not the failure.
The environment built around the model was.
That distinction matters because I suspect we are thinking about AI in the wrong category. We continue to speak of it as though it were primarily a tool. History suggests something different. The most consequential technologies eventually become environments.
Agriculture did more than increase food production. It reorganized how people lived, how families were structured, how labor was divided, and how civilizations formed. The printing press did more than reproduce books. It changed memory, authority, education, and religion. Social media did more than lower the cost of communication. It reshaped attention, friendship, and the social environments in which identities developed.
Artificial intelligence will almost certainly do the same. Its greatest effect may not be the answers it produces but the cognitive environment it creates for the people who rely on it every day.
Anthropologists have long argued that human beings are shaped not only by what they believe but by the environments they inhabit and the practices they repeat. We gradually acquire habits of attention, judgment, and interpretation that reflect the worlds around us. In Pierre Bourdieu’s language, our habitus is formed through repeated interaction with our environment. Whether or not one accepts his broader theory, the underlying insight is difficult to deny: people adapt to the worlds they repeatedly inhabit.
Technology therefore never remains external to us. It slowly becomes part of the process by which we become who we are.
Seen from that perspective, the Wharton study is not simply an education study. It is evidence that different AI environments produce different cognitive outcomes, even when the underlying model is essentially the same. One tutor preserved productive struggle. The other quietly removed it. The important variable was not intelligence. It was the developmental environment.
The same pattern is beginning to appear elsewhere. A recent study following nearly twenty-seven thousand secondary school students in China found that students who increasingly relied on AI to complete homework earned higher homework marks but performed substantially worse on closed-book examinations, with the largest declines among the strongest students. The mechanism appears remarkably consistent. When the machine performs more of the cognitive work, the human performs less.
None of this is inevitable. The difference between the two tutors was not capability. It was design.
A model can draft the memo for a junior analyst, or it can coach her through drafting it herself. It can answer every question immediately, or it can preserve the productive struggle through which judgment is formed. Those choices may seem like interface decisions, but they are really decisions about the kind of cognitive environment we want to create.
This is where I think our conversation about AI safety needs to expand.
Much discussion of artificial intelligence assumes that as intelligence becomes abundant, many of today’s constraints will disappear. History suggests something more subtle. Every technological revolution removes some constraints while creating others. Electricity made light abundant but transformed how people used time. The internet made information abundant, making attention comparatively scarce. AI will make answers abundant. The scarce resource will increasingly be the human capacity to ask worthwhile questions, recognize when familiar models no longer describe reality, distinguish genuine understanding from fluent retrieval, and construct meaning rather than simply consume it.
These are often described as “soft skills.” I think that misses the point. They are becoming the binding constraint. In a world where almost anyone can obtain an answer instantly, the comparative advantage shifts to the person who knows which questions are worth asking and when the answer, however fluent, is incomplete.
That is why I believe AI safety cannot be defined solely by the behavior of the model. It must also include the long-term effects of the environments those models create.
Hassabis is right that civilizational technologies require civilizational institutions. But those institutions should evaluate more than whether frontier models remain aligned with human intentions. They should also ask whether prolonged interaction with those systems strengthens or weakens the distinctly human capacities that will matter most once intelligence itself becomes abundant.
The central challenge of AI is not simply building intelligent machines. It is deciding what those machines make of the people who use them. That decision is already being made, one design choice at a time. The only question is whether we make it deliberately—or simply inherit it by default.
My book, The Adaptability Quotient, is out September 15, 2026. It’s a framework for thinking clearly and deciding well when the ground keeps shifting. CEO of Citadel Ken Griffin calls it essential reading for anyone seeking to sharpen judgment and thrive amid constant change. Pre-order at www.theaqbook.com.