Recent AI fears are exposing a critical governance dilemma

Recent AI fears are exposing a criticial governance paradox: what the industry wants and how it acts strike at the heart of what can sometimes be a vicious circle of strategy. 

We’ve heard fears around AI before, but in September 2026, the latest comments from industry insiders seemed to strike a chord worldwide. What they said ranged in severity, but the words we’ll all come away with are undoubtedly those of Jacob Coxon – former researcher at both OpenAI and Anthropic – who said he earnestly believes AI could “kill us all by the end of the decade”. 

This article isn’t here to prove whether he’s right or wrong; it’s here to explore the governance paradox underscoring it – a dilemma we’ve seen before in many different contexts, but it takes on a new level of importance with an innovation like AI. 

To understand it, we need to understand what comments like AI “killing us all” mean in practice. 

Some critics argue that these words – and similar concerns from other insiders – are just part of a larger effort around the long-term value of the AI industry, with various degrees of intent involved. Time will give a proper indication of how true that is. 

But for the moment, let’s assume that it’s all coming from real concerns around AI’s capabilities, our understanding of them, and our ability to “pull the plug” if something goes wrong. 

These fears have prompted the leaders of prominent AI firms – Sam Altman of OpenAI and Elon Musk of xAI  – to join with Anthropic head Dario Amodei in calling for a slowdown in AI development in the name of security and control. 

Right here is where the dilemma starts, and it becomes apparent when we put these three companies under the microscope and compare words and actions. 

Right now, the AI company bosses are hovering between proposing a general slowdown and outright calls for new regulation. In other words, there’s appetite for moderation. However, did you notice that this appetite is mainly generalised calls related to the entire industry, not their own desires for self-moderation?

The underlying issue is that there are two underlying governance responsibilities pulling against each other. 

On one side is the argument that sees AI as a legitimate threat if not managed properly. This side is rooted in the belief that things are currently moving too fast for vital control to be maintained. In this era, more than ever, we can understand how critical this concern is, because it raises entirely legitimate fiduciary concerns around ethics and stakeholder impact. Continuing on the current path may literally put people in danger. 

Then, on the other side, we see the competition element. The biggest AI firms remain in a race against each other to develop the most advanced products (not to mention the broader race that lawmakers, including Donald Trump, say exists with China). From this angle, there’s also a perfectly legitimate governance argument to keep moving, to seize on marketplace opportunities, and to make decisions with such success in mind. 

The conflict between the two gives us the critical governance dilemma: a vicious cycle of priorities that’s hard to escape in a hurry. It’s what gives rise to the frankly bizarre scenario where industry leaders will go to the press and shout “please stop us, because we won’t stop ourselves”.

The dilemma is not new. Since ethics and stakeholder capitalism first entered boardroom and C-suite agendas decades ago, we’ve seen the cycle rear its head: firms with expert insights recognise potential dangers, but choose not to act until they’re told to, because anything earlier would prompt shareholder/investor anger. 

We’ve seen it with geopolitics, especially as it’s become more fraught in the 2020s. Often forced to pick sides in global conflicts like Ukraine, Iran or Gaza, companies will deliberately avoid it in business practice until regulations, sanctions or dramatic public outcry force them to. 

Climate change is another example: Firms have, for years, based their strategies around what they’re told to do by outside pressure rather than internal principles. It’s why we’re seeing such divergence between US companies (under a regime far more hostile to emissions reductions and ESG principles), and European companies (increasingly bound by sustainability reporting requirements).