Raw capture — raw/its-time-to-investigate-the-ai-labs.md

url: https://calnewport.com/its-time-to-investigate-the-ai-labs/ date_fetched: 2026-09-29

Over the last several months, the two leading frontier AI labs have shown some brazen behavior.

It started with a series of carefully planned announcements and reports from OpenAI that try to establish how unnerving and powerful (not to mention felonious) their LLM-powered agent systems have become.

They then handed the baton to Anthropic, whose employees began publicly debating the exact probability that these technologies would lead to human extinction. They delivered this all with an eerie calmness that conveyed a nihilistic inevitability about their work.

The ground suitably softened, the campaign crescendoed with Anthropic CEO Dario Amodi releasing a letter, titled “We Must Pace the Frontier,” in which he enumerates all the harms his own company’s research might cause, and then, instead of apologizing and promising to stop, concludes that catastrophe can only be avoided if we “build the technology in the right way,” which involves – surprise, surprise – the government slowing down potential competitors and allowing the labs to take the lead in advancing the relevant tech. OpenAI CEO Sam Altman quickly tweeted his support for this brave plan.

Internally, these labs have long embraced this style of messianic thinking, casting themselves as humanity’s only hope against the defied powers of superintelligent AI. This summer, in some sense, was their attempt to induct the rest of us into this long-held ideology.

It didn’t work.

Instead of inspiring the masses to stand up and applaud these engineers’ heroic rationalism, it made us instead start to ask, “What the hell is going on over in those labs?”

To that end, last Thursday I published an op-ed in The New York Times calling on Congress to begin a public fact-finding mission to uncover what research projects OpenAI and Anthropic are conducting, how they’re conducting them, and for what purposes.

I suggested three key areas to explore with this questioning:

• We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems. The frontier labs want us to think of AI as a singular technology on an inevitable, fixed trajectory. In reality, most of the problems in recent months result from a narrow band of incautious experiments, conducted mainly by the frontier labs. They need to justify why they’re running them.
• We must examine the internal safety procedures surrounding these research efforts. OpenAI revealed a long series of unauthorized hacking attacks by their autonomous agents. Why weren’t these efforts stopped after the first such incident was revealed? Why not pursue criminal liability for them running systems they knew would likely commit crimes?
• We have to investigate the role of apocalyptic futurist ideologies in the decisions being made by the frontier labs, both about what research to pursue and how fast to pursue it. I’m increasingly concerned that OpenAI and Anthropic, in particular, are becoming increasingly reckless because they believe collateral damage is justified in a race to redeem humanity. (For more on these beliefs and their connections to the labs, see: this and this and this, all published in the Timesin just the last several weeks.)

“We need to stop letting a small number of private companies, acting and talking in increasingly erratic ways, dictate how we’re supposed to feel about A.I.,” I concluded. “We’ve heard what they want to say; now it’s Congress’s turn to step in on our behalf to find out what’s really going on.”

Love these articles, and thank you for them. I suspect a typo in the phrase “the defied powers of superintelligent AI” was meant to be “deified”.

Jensen Huang’s interview with Ezra Klein put this clearly: Labs asking for government to slow them down is abdication of responsibility and blame. They are responsible to make their products safe. No one is pressuring them to release unsafe products.

They need to likely grow their investment and headcount 10x, and 80% should be safety and testing.

And finally, if the labs are unable to stop releasing unsafe products, they should probably be shut down

Everyone should listen to that interview. It’s a window into how these folks think and it’s frankly shocking. In Huang’s case, I assume his odd answers and lack of concern over AI safety are the result of being compromized to the tune of a 4 trillion dollar market capitalization. His reasoning lacked coherence and felt uninformed (especially when it came to the issue of alignment and safety). Huang seems completely unwilling to entertain that intelligent systems (even generally intelligent systems) are anything but “software systems”.

I have no idea why he agreed to do that interview. It strongly reinforces the concluding statement in the essay above.