Thursday, 17 September 2026
The next major cybersecurity failure may look very different from the breaches companies have spent decades preparing for. Instead of an attacker stealing an employee’s password and entering a corporate system, the risk could increasingly come from an authorised AI system taking an incorrect, unexpected or excessive action using access it was legitimately given.
That shift is becoming particularly important as businesses move from generative AI systems that primarily answer questions towards AI agents capable of interacting with software, accessing data, initiating workflows and eventually making changes to operational systems. Speaking at Ai4 2026 in Las Vegas, Sahil Yadav, Head of Software and AI at Applied Optoelectronics, argued that the central challenge is no longer simply whether an AI model performs well in testing. Companies increasingly need to understand what happens when the model is connected to a live environment and given the ability to act.
The issue is particularly significant in telecommunications, utilities, industrial facilities and other areas of critical infrastructure. An incorrect AI-generated answer in an office application may be inconvenient. An incorrect automated decision affecting a communications network, industrial process or safety system can have substantially larger consequences. Yadav’s argument was that companies should therefore think about trustworthy AI as an operational engineering problem rather than simply a model-quality problem. Explainability, traceability, controls and human intervention become part of the infrastructure surrounding the AI.
This distinction will become more important as the number of non-human identities inside corporate systems expands. Traditional identity security has primarily been designed around employees, contractors and administrators. Modern businesses already contain large numbers of service accounts, APIs, automated workloads and machine identities. AI agents add another category because they may not only access information but interpret situations and decide what action to take.
The potential security problem is therefore not necessarily that an AI agent has obtained unauthorised credentials. The agent may have completely legitimate access while behaving in a way that the organisation did not anticipate. This changes the meaning of access control. Giving an employee permission to perform a task generally assumes that the individual will exercise judgement before acting. An autonomous system can operate much faster and potentially repeat an incorrect action across thousands of systems or transactions before anyone recognises what has happened.
The emerging question is consequently not only who or what has access, but what that identity is permitted to do, under which conditions, for how long and within what limits. Companies are beginning to consider human identities, conventional machine identities and AI agents within a broader security environment, reflecting the growing concern that autonomous software requires its own access lifecycle and controls.
For critical infrastructure operators, the problem extends beyond conventional cybersecurity. AI can also make operational errors without being compromised by an attacker. Telecommunications networks increasingly use machine learning and automation for fault detection, predictive maintenance, network optimisation and other operational functions. As these systems become more autonomous, an incorrect inference can potentially move from being a recommendation on a screen to an action affecting live infrastructure.
The same principle applies elsewhere. A sensor can provide inaccurate information. Data can drift away from the conditions under which a model was developed. An AI system can interpret a situation incorrectly. Individually, each component may appear to be operating normally while the resulting automated decision is wrong.
These failures are particularly difficult because they can remain invisible for long periods. Yadav described them as silent failures. The system continues functioning and may continue producing apparently legitimate outputs, but the quality of the underlying decision has deteriorated.
This makes traceability critical. When an autonomous system takes an action, companies need to know what information it received, which model or agent was involved, what decision was proposed, which systems were accessed and whether a human authorised the final step. Without that history, investigating an AI-driven incident can become substantially more difficult than tracing a conventional software failure.
Auditability may therefore become one of the defining requirements of enterprise AI. The objective is not necessarily to understand every mathematical operation inside a large model. Organisations need enough operational information to reconstruct why a decision was made and determine whether the system was behaving within its intended boundaries.
Yadav identified explainability, traceability and guardrails as three important components of this approach. Explainability means providing operators with enough information to understand why a system reached a conclusion. The explanation also needs to be useful to the person responsible for the system rather than understandable only to a data scientist. Traceability creates the operational record showing what happened, while guardrails establish the limits of autonomous behaviour.
Those limits can take many forms. An AI agent might be permitted to diagnose a network problem but not change network configuration. Another system might be permitted to implement changes only below a predetermined financial or operational threshold. Higher-risk decisions might automatically require human approval.
This creates what Yadav described as adaptive control, where the degree of autonomy can increase as the organisation develops confidence in a particular system. Instead of choosing between complete automation and complete human control, companies can introduce autonomy gradually. An agent might initially operate only in an advisory role. It could later receive permission to perform a small number of reversible actions before eventually gaining greater operational responsibility.
Human oversight consequently remains central. The relevant distinction is increasingly between having a human in the loop, where approval is required before an action takes place, and having a human on the loop, where the autonomous system operates but remains under human supervision. The appropriate model will depend heavily on the potential consequences of failure.
An automated recommendation affecting a low-value internal workflow can reasonably operate with relatively limited oversight. An agent capable of modifying telecommunications infrastructure, financial transactions or industrial safety systems requires much stricter controls.
Yadav’s previous experience with industrial safety technology provides an example of why this matters. At connected-worker technology company Guardhat, where he previously worked, location and telemetry information formed part of systems designed to identify potentially dangerous conditions around industrial workers.
Yadav described a case in which inaccurate GPS information generated large numbers of false safety alerts. According to his account, approximately 70% of the alerts were false, eventually creating a situation where workers began ignoring warnings because so many were incorrect. The figure represents the specific case described by Yadav rather than a general performance measure for the technology.
That represents one of the most dangerous forms of automation failure. The technology does not simply produce incorrect information; repeated false alarms can alter human behaviour until users lose confidence in the system itself.
Yadav said the underlying problem took approximately eight months to resolve because the system lacked sufficient explainability. He argued that better telemetry and explanation could have exposed the GPS drift considerably earlier and allowed engineers to address the problem before confidence in the alert system deteriorated.
The lesson extends well beyond industrial safety. If an AI-powered cybersecurity system continuously produces false threats, security teams may eventually ignore it. If a predictive-maintenance system repeatedly generates unnecessary interventions, engineers may stop trusting its recommendations. If an autonomous financial-control system frequently blocks legitimate transactions, employees may look for ways around it.
Trust therefore has an operational value. Yadav suggested that companies should measure not only whether an AI system is accurate but how quickly the organisation can identify and correct an AI-related error. He described a measure centred on the time required to resolve errors that require explanation and investigation. The broader principle is useful even if companies use different terminology: the longer an autonomous error continues before detection, the greater its potential impact.
A second consideration is the consequence of being wrong. An AI error that requires an employee to rewrite an email has negligible organisational risk. An incorrect credit decision, network configuration or industrial-safety warning can create financial, regulatory and reputational consequences. Companies therefore need to evaluate AI systems according to both the probability of failure and the damage that failure could cause.
That could become increasingly important at board level as companies deploy autonomous technology. Conventional AI performance measures such as model accuracy or response latency say relatively little about the financial exposure created when an agent is allowed to take action.
Cybersecurity architecture will also need to adapt. AI agents should increasingly be treated as identities in their own right, with defined owners, permissions, credentials, activity records and expiry or revocation procedures. Access should follow least-privilege principles rather than granting an agent broad access simply because it needs to interact with several corporate systems.
The identity also needs to remain connected to human accountability. A business should be able to identify who created an agent, who authorised its permissions, which business function owns it and who can disable it.
This becomes increasingly complicated when agents begin communicating with other agents. An employee may ask one AI assistant to perform a relatively simple task, while that system delegates parts of the work to several specialised agents using different databases, external services and corporate applications. The resulting chain of machine identities can be far more difficult to monitor than one employee operating one application.
Security teams will therefore need visibility across the entire sequence of activity. The expansion of AI agents could consequently push cybersecurity away from static access management towards continuous behavioural control. Authentication will remain essential, but proving that an identity is legitimate does not prove that every action it performs should be allowed.
That distinction may become one of the defining security problems of the agentic AI era. The largest risk may not be a malicious AI system deliberately attacking its owner. It may be a perfectly authorised system operating at machine speed, using valid credentials and following an objective in a way that its designers failed to anticipate.
For companies deploying AI inside critical infrastructure, the answer is unlikely to be slowing adoption altogether. It is to design autonomy around observable actions, restricted permissions, clear accountability, reversible decisions and human intervention when the potential consequences become too large.
The security question is therefore changing. For decades, organisations have asked whether the person logging into a system is really who they claim to be. As autonomous AI spreads through corporate infrastructure, they will increasingly have to answer another question: even if the identity is legitimate, should the machine be allowed to do what it is about to do?
Source: CIJ.World Research & Analysis Team
