AI Existential Risk Deserves Our Attention

My first reaction when I read that an Anthropic researcher had quit and warned that advanced AI could potentially wipe out humanity was pretty simple:

Oh shit.

That seems like a reasonable response when someone who actually helps build these systems says the people inside AI labs believe the technology could become dangerous enough to threaten humanity.

Then comes my second reaction.

Stay calm.

One of the values I try to live by is calm in the midst of chaos. Fear makes us stupid. So does blind optimism. This moment calls for attention, curiosity, judgment, and a willingness to take the people closest to the technology seriously.

The warning is coming from inside the building

Jacob Coxon recently resigned from Anthropic after working on AI research at both Anthropic and OpenAI. He warned that AI companies are moving toward systems capable of helping improve future AI systems, potentially creating a cycle of increasingly powerful artificial intelligence.

His concern is what researchers call recursive self-improvement  – essentially AI programming and improving itself. Maybe one day it’ll see us as a roadblock? Even as a Computer Engineering Student at ASU, I’d think, “Jeez, when these computers can program themselves, we’re – humans – screwed.”

Coxon said people building these systems genuinely believe advanced AI could pose an existential threat to humanity. Anthropic alignment researcher Evan Hubinger publicly supported much of that concern and said Anthropic does not yet have a clear solution for aligning superintelligent AI.

That deserves our attention.

These researchers also operate inside a very specific world. They spend their days thinking about models, capabilities, alignment, benchmarks, security and increasingly powerful machines. Their perspective matters enormously, but it is still a perspective shaped inside the walls of AI labs.

This conversation needs a much bigger room.

The companies themselves are starting to say it

OpenAI recently made a significant public statement about this same issue.

The company said fully autonomous recursive self-improvement is not happening today and should not be pursued until it can be done safely. OpenAI also called for common safety standards, meaningful human control, and clear thresholds that could require AI development to slow or stop.

OpenAI has also said AI already speeds up parts of the research used to build future models. The company says its goal is to develop automated AI researchers that remain under human supervision.

That got my attention.

So this conversation has moved beyond a researcher with a scary prediction.

The organizations building frontier AI are publicly discussing control, catastrophic risk, self-improvement and conditions under which progress may need limits.

We should probably listen.

AI safety is becoming a community issue

Most of my work with AI focuses much closer to home.

I worry about people handing too much of themselves over to AI. I call that soul drift.

You slowly give away your writing, thinking, judgment, creativity, relationships and expertise. Eventually AI starts sounding a lot like you because you have started sounding a lot like AI.

That matters.

This new conversation goes several miles beyond soul drift.

We are talking about who controls increasingly capable systems and who gets to decide how far they go.

That makes AI a business issue, a community issue, a political issue and eventually an international issue.

We need more people at the table

I don’t believe government alone can keep pace with this technology. The companies building frontier AI understand it far better and move far faster.

I think the major AI labs need to help form a coalition that includes people far beyond the labs themselves:

• AI researchers and safety experts
• Economists and business leaders
• Governments and international partners
• Scientists, educators and community leaders
• Ordinary citizens who have to live in the world these systems create

China needs to be part of that conversation too.

We may never get every country, company or individual to cooperate. That cannot become an excuse for doing nothing.

We can establish standards. We can monitor frontier development. We can create independent oversight. We can decide what capabilities should reach the public. We can create pressure when someone crosses agreed boundaries.

Some kind of governor has to exist.

Pay attention

I still use AI every day.

I still believe it can help businesses, nonprofits and communities do extraordinary things. I see that happen constantly.

I also believe being human means living, loving, laughing, struggling, suffering, connecting, creating and caring about each other.

Those things matter even more as the machines become more capable.

So my response to the growing discussion about AI existential risk is pretty simple.

Pay attention.

Read beyond the headlines. Listen to the researchers. Question them too. Stay connected to other people. Talk about what kind of future we actually want.

And stay calm in the midst of chaos.

Because whatever happens next with AI, humanity needs to remain an active participant in the decision.

If you’d like a place to keep learning and talking about these issues without drowning in AI hype, that’s part of what we do in the Cingularis AI Lab.

AI Solutions with Soul