As the EU Cyber Resilience Act and UK Cyber Security and Resilience Bill push organisations to scrutinise supplier risk and operational resilience, AI adoption may be creating a new concentration risk that many businesses cannot see.
OpenAI’s chief scientist Jakub Pachocki recently called for “extreme caution” over the pace of AI development, warning that organisations and governments are not prepared for where increasingly capable systems could take us. He is not alone. Former OpenAI and Anthropic researcher Jacob Coxon has warned about the race towards self-improving AI, while Anthropic CEO Dario Amodei’s subsequent intervention drew support from other leading technology figures, including Elon Musk.
The mounting concern is that increasingly autonomous systems could progress towards superintelligence faster than our ability and political willingness to control them, as governments and technology companies race to develop ever-more-powerful models.
For CISOs, there is a much more immediate resilience question. How well do they understand the AI capabilities their businesses are rapidly becoming dependent upon?
Organisations may use several AI services and believe their risk is diversified, when those services ultimately rely on the same foundation models, hyperscalers and infrastructure. What looks like diversity can therefore mask common points of failure.
The real resilience test is whether the business can keep operating if a critical AI capability becomes unavailable.
Regulation is pushing organisations to look underneath
The regulatory direction of travel is already towards much greater scrutiny of technology supply chains.
The EU Cyber Resilience Act (CRA) and the UK’s Cyber Security and Resilience Bill approach the problem from different directions. The CRA largely comes at resilience from the product side. From 11 September, manufacturers selling relevant connected products into the EU have just 24 hours to report actively exploited vulnerabilities, potentially before they have the complete picture.
The UK Bill comes at resilience more from the operator and supply-chain side, with greater emphasis on critical suppliers and the risks created when essential services depend upon third parties. There has also been debate around AI as the Bill progresses, particularly over where responsibility should sit between organisations using AI and the companies developing it.
For CISOs, that distinction does not change the immediate problem very much. If your business depends upon an AI capability to operate, you need to understand the risk whether the regulation explicitly calls it an AI provider or not.
The two regimes create different obligations, but the underlying resilience challenge increasingly needs to be managed as one. AI needs to become part of that same conversation.
The problem is that regulation is struggling to keep pace with the capabilities now being developed. With the US prioritising acceleration in its competition with China and meaningful international consensus still distant, frontier AI companies may ultimately have to set tougher safety standards themselves.
Five AI suppliers might still mean one dependency
Most organisations have become much better at assessing direct suppliers, looking at security controls, incident response, data processing and increasingly ownership and jurisdiction.
But AI makes those dependencies more complicated.
An organisation might use five AI-powered services from five different vendors. On paper, concentration risk looks relatively low. But perhaps four use the same foundation model, three run on the same hyperscaler and all five rely on infrastructure controlled within the same jurisdiction.
Suddenly, five suppliers do not look particularly diversified at all.
We have already seen this elsewhere in technology supply chains. Moving three workloads to three different services achieves very little if all three ultimately depend on the same hyperscaler, identity provider or jurisdiction.
CISOs therefore need to look beyond the direct AI supplier and understand which foundation models and infrastructure critical services depend upon, and where apparently separate services share the same points of failure.
Otherwise, organisations risk diversifying their suppliers while concentrating the actual risk.
AI sovereignty is about more than where the data sits
This also moves the sovereignty conversation on.
For years, digital sovereignty has focused heavily on where data is stored and processed and which jurisdiction applies. Those things still matter, but AI sovereignty increasingly needs to be about who ultimately controls critical capability.
Who controls the model and access to it? Where is the provider based and under which laws does it operate? And, crucially, what does the exit look like if that provider suddenly becomes a problem?
A critical AI service could become unavailable because of a cyber incident or technical outage, but disruption could equally come from a commercial decision, regulatory intervention or geopolitical change.
Nobody wants to get six months into embedding an AI platform across critical operations only to discover that circumstances have changed and they now need to work out how to remove it.
What happens if the AI stops?
The most useful resilience test is to ask what would actually happen if a critical AI service became unavailable tomorrow.
Could people still perform the process? Could another provider take over? Could the organisation retrieve its data and move the workload? And has anybody actually tested whether the alternative works?
An exit clause written into a contract is useful, but it is meaningless if the organisation has never tested whether it can actually migrate, rebuild or operate the replacement.
Identifying a concentration risk is one thing. Working out how to remove it without disrupting an essential business service is something else entirely.
Skills are part of the dependency too
There is another risk here that is much harder to capture on a supplier map.
As organisations become more reliant on AI-assisted coding, analysis, security operations and decision-making, some of the underlying internal capability could gradually erode.
If employees have relied on AI to perform a process for several years, can they still operate effectively without it? If developers become heavily dependent on AI-assisted coding, does the organisation retain enough internal knowledge to migrate, rebuild or recover a critical system?
This is not an argument against AI adoption. But there is little value in having a technically credible exit strategy if nobody has the skills to execute it.
The objective should be optionality
Organisations do not need complete independence from AI providers. For most businesses that would be unrealistic and potentially damaging to their ability to compete.
The objective should be optionality.
Map the AI capabilities supporting critical business services and then look beyond the direct supplier. Understand the models, cloud platforms and infrastructure those services rely upon and identify where different providers share common dependencies.
For the most important services, determine whether credible alternatives exist and whether they could actually be activated under pressure. Organisations also need enough internal capability to operate while that happens.
Resilience is not achieved through procurement clauses alone. You need people who understand the systems well enough to operate, migrate, rebuild and recover them.
Make AI part of the resilience conversation
There is a danger that organisations treat CRA, the Cyber Security and Resilience Bill, AI governance, supplier assurance and sovereignty as separate compliance exercises.
The differentiator will be whether CISOs can turn them into one coherent resilience conversation at board level.
What could cause downtime? What would that downtime cost us? Where have we concentrated too much dependency? And if access to a critical AI capability disappeared tomorrow, what would actually happen?
Because ultimately that is what boards need to understand. Not another regulatory acronym or AI governance framework, but whether the business can keep operating when technical, commercial, regulatory, geopolitical or cyber risk suddenly makes a critical capability unavailable.
The real resilience test is not how widely AI has been deployed or how many different providers appear on the supplier register. It is how well the business can operate if that capability is disrupted.
Daryl Flack
Daryl Flack is Partner at Avella Security. A highly experienced and trusted cyber security professional with over 25 years dedicated to protecting the UK’s national interests, Daryl works at the forefront of securing the UK’s Critical National Infrastructure (CNI) and is widely respected for his integrity, leadership, and strategic insight.
