Deepfakes and the New Enterprise Trust Problem
Introduction
The traditional logic of workplace security is built on the baseline assumption that an employee would be able to identify a suspicious request by form and/or sound. The rollout of AI-generated voice and video is steadily dismantling this assumption. The authenticated voice of a familiar executive or the face of a colleague in a video meeting is no longer a guarantee of variety. This paper reviews recent survey evidence of deepfake-enabled social engineering, briefly considers the Arup fraud as a case study and reviews Gartner's and United States government agency guidance. It argues that organisations should stop relying on employees to spot fakes and instead build independent verification, limited authority and escalation into the everyday workflows that move money and grant access, long before a convincing fake can do damage.
The Scale of the Threat
Gartner revealed in September 2026 that the survey (of 297 senior cybersecurity leaders, March to May 2026) found 41% had been subject to at least one deepfake social engineering attack during an audio employee call in the last 12 months; 36% of deepfake social engineering attacks happened on a video call with an employee in the last 12 months. Gartner concluded that AI makes social engineering more prolific, personalised and believable, while it makes common detection signals less dependable.
The survey also highlights the continued use of older methods. 79% of respondents reported at least one email phishing, spear-phishing, or business email compromise incident, and 58% vishing or smishing. Deepfakes are not replacing tried-and-true techniques but supplementing them. Craig Porter, a director analyst at Gartner, said, "Attackers can combine phishing, business email compromise, synthetic media and the aggregation of personal context across multiple channels, yet most attacks still rely on compromised credentials, simple account takeover and poor recovery processes."
Why it matters: Ability to receive requests can now be routed through the same communication systems that high-trust messages go through. With information gathered from the web, as well as synthetic voice and video, fraudsters can impersonate executives, vendors, suppliers, colleagues, or customers.
Why Spotting the Fake Is Not Enough
The 2024 attack on Arup is an example of this. A finance staff member in an engineering firm in Hong Kong took part in a video conference in which he thought he was speaking to the firm's CFO and others, but all participants were deepfakes. The employee initially suspected a phishing email, but he put aside his doubts after the video call because the participants looked and sounded like colleagues he recognised. He then approved 15 payments of a total HK$200m (US$25.6m). Arup confirmed the voices and faces were fake, though its internal cybersecurity was not breached, and its global CIO, Rob Greig, said attacks of this kind had increased substantially in recent months. The firm's East Asia chairman said staff were warned of increasing attacks of this kind around the world.
This case is instructive because the control that failed was human judgement with a plausible performance. The employee's first reaction was right; it was a video call that overruled this. Detection by the request recipient was the only control.
Government guidance points the same way. A joint information sheet from the NSA, FBI and CISA warns that the most substantial synthetic media threats include impersonating leaders and financial officers and using fraudulent communications to gain access to networks and sensitive information. It advises organisations to have a response plan and to consider technologies that detect deepfakes and establish the provenance of media. Detection tools have a place, but they are one layer, not a foundation.
Building Verification into Workflows
Achyuth Krishna, Head of IT, Information Security and Procurement at Whatfix, says that the real question is no longer 'Can the employee tell that the message is generated by AI?' But whether your organisation has designed its workflow so that a single deceptive interaction cannot become a security incident. "Trust itself has now become a new attack surface," he says.
Gartner's suggested mitigations echo this sentiment. "Porter recommends that CISOs evaluate AI-led social engineering risks with the same level of scrutiny as identity and access risks," the analyst firm said. Among the suggestions that the firm made to organisations that want to defend against AI-led social engineering attempts include establishing secure verification as standard operating procedure for any important request, by whatever means they are communicated, by email, voice, video, collaboration platform or AI system, and using trusted channels of verification as well as stronger authentication for account recovery, privileged access, and payment approval. In terms of design, this suggests a few principles:
1. Before deciding to authorise payments, change privileged access or disclose information, make sure that you get independent confirmation from a different source.
2. This should apply to everyone, including requests that appear to come from a known senior executive, so that employees never need to challenge authority personally.
3. Clearly outline situations where your team needs to stop, confirm, and escalate.
4. Test these procedures in scenarios that are as close to reality as possible rather than just training on an annual basis.
In this case, a simple video call would not have been enough to trigger the release of the money. Validation would have taken place outside the attacker's communication channel.
Extending Identity to AI Agents
As companies open up apps and business data to software agents, Krishna says those agents need their own identities, limited rights, and audit logs. Gartner has also identified that compromised AI apps, prompt injection, and agents acting outside of their bounds are threats that security teams will need to handle.
The logic is the same as in the human case. An agent, given virtually unlimited power, can be misused in exactly the same way that a human employee can be. Seen as identities to be verified, constrained and observed rather than as trusted, remoting extensions of their original creators, agents preserve the principle of limited authority.
Conclusion
Deepfakes have made impersonation cheap, personal and persuasive, and many organisations have already been hit. Detection will remain useful, but it cannot carry the burden alone. The stronger response is structural: independent verification for consequential requests, narrow permissions for people and agents, and regular testing of escalation procedures. The aim is to ensure that no single call, message or agent holds enough unchecked authority to turn a successful deception into a financial loss or a data breach.
Sources
- Tech Observer: Deepfakes are turning trust into a new enterprise security risk
- Gartner: Survey Finds 41% of CISOs Reported At Least One Social Engineering Incident Involving a Deepfake in the Past 12 Months
- Help Net Security: The latest deepfake numbers give CISOs plenty to worry about
- NSA: NSA, U.S. Federal Agencies Advise on Deepfake Threats
- CNN: British engineering giant Arup revealed as $25 million deepfake scam victim (syndicated copy)
Related Blogs
.webp)
Introduction
The rapid advancement of technology, including generative AI, offers immense benefits but also raises concerns about misuse. The Internet Watch Foundation reported that, as of July 2024, over 3,500 new AI-generated child sexual abuse images appeared on the dark web. The UK’s National Crime Agency records 800 monthly arrests for online child threats and estimates 840,000 adults as potential offenders. In response, the UK is introducing legislation to criminalise AI-generated child exploitation imagery, which will be a part of the Crime and Policing Bill when it comes to parliament in the next few weeks, aligning with global AI regulations like the EU AI Act and the US AI Initiative Act. This policy shift strengthens efforts to combat online child exploitation and sets a global precedent for responsible AI governance.
Current Legal Landscape and the Policy Gap
The UK’s Online Safety Act 2023 aims to combat CSAM and deepfake pornography by holding social media and search platforms accountable for user safety. It mandates these platforms to prevent children from accessing harmful content, remove illegal material, and offer clear reporting mechanisms. For adults, major platforms must be transparent about harmful content policies and provide users control over what they see.
However, the Act has notable limitations, including concerns over content moderation overreach, potential censorship of legitimate debates, and challenges in defining "harmful" content. It may disproportionately impact smaller platforms and raise concerns about protecting journalistic content and politically significant discussions. While intended to enhance online safety, these challenges highlight the complexities of balancing regulation with digital rights and free expression.
The Proposed Criminalisation of AI-Generated Sexual Abuse Content
The proposed law by the UK criminalises the creation, distribution, and possession of AI-generated CSAM and deepfake pornography. It mandates enforcement agencies and digital platforms to identify and remove such content, with penalties for non-compliance. Perpetrators may face up to two years in prison for taking intimate images without consent or installing equipment to facilitate such offences. Currently, sharing or threatening to share intimate images, including deepfakes, is an offence under the Sexual Offences Act 2003, amended by the Online Safety Act 2023. The government plans to repeal certain voyeurism offences, replacing them with broader provisions covering unauthorised intimate recordings. This aligns with its September 2024 decision to classify sharing intimate images as a priority offence under the Online Safety Act, reinforcing its commitment to balancing free expression with harm prevention.
Implications for AI Regulation and Platform Responsibility
The UK's move aligns with its AI Safety Summit commitments, placing responsibility on platforms to remove AI-generated sexual abuse content or face Ofcom enforcement. The Crime and Policing Bill is expected to tighten AI regulations, requiring developers to integrate safeguards against misuse, and the licensing frameworks may enforce ethical AI standards, restricting access to synthetic media tools. Given AI-generated abuse's cross-border nature, enforcement will necessitate global cooperation with platforms, law enforcement, and regulators. Bilateral and multilateral agreements could help harmonise legal frameworks, enabling swift content takedown, evidence sharing, and extradition of offenders, strengthening international efforts against AI-enabled exploitation.
Conclusion and Policy Recommendations
The Crime and Policing Bill marks a crucial step in criminalising AI-generated CSAM and deepfake pornography, strengthening online safety and platform accountability. However, balancing digital rights and enforcement remains a challenge. For effective implementation, industry cooperation is essential, with platforms integrating detection tools and transparent reporting systems. AI ethics frameworks should prevent misuse while allowing innovation, and victim support mechanisms must be prioritised. Given AI-driven abuse's global nature, international regulatory alignment is key for harmonised laws, evidence sharing, and cross-border enforcement. This legislation sets a global precedent, emphasising proactive regulation to ensure digital safety, ethical AI development, and the protection of human dignity.
References
- https://www.iwf.org.uk/about-us/why-we-exist/our-research/how-ai-is-being-abused-to-create-child-sexual-abuse-imagery/
- https://www.reuters.com/technology/artificial-intelligence/uk-makes-use-ai-tools-create-child-abuse-material-crime-2025-02-01/
- https://www.financialexpress.com/life/technology-uk-set-to-ban-ai-tools-for-creating-child-sexual-abuse-images-with-new-laws-3735296/
- https://www.gov.uk/government/publications/national-crime-agency-annual-report-and-accounts-2023-to-2024/national-crime-agency-annual-report-and-accounts-2023-to-2024-accessible#part-1--performance-report
.webp)
Executive Summary
As Tamil Nadu voted in the 2026 Assembly elections, with 84.69 percent polling recorded on April 23, a purported cover page of Frontline magazine began circulating on social media. The viral image featured a massive rally crowd of South Indian actor and Tamilaga Vetri Kazhagam (TVK) chief Joseph Vijay, claiming that a “Vijay Wave” had emerged in the state. The alleged cover also stated that Tamil Nadu was witnessing a new political force after five decades, one that had challenged the dominance of Dravida Munnetra Kazhagam (DMK) and All India Anna Dravida Munnetra Kazhagam (AIADMK).
However, research by the CyberPeace Research Wing found that the viral cover page is fake. Frontline magazine has not published any such edition, and its latest issue is related to health.
Claim:
X user “Dr. Ravishankar Sadasivam” shared the viral image on April 22, 2026, claiming the “wave is real” and suggesting that Vijay could emerge as a top contender in a three-cornered contest. The post further claimed that after MGR, Vijay was drawing the largest spontaneous crowds in Tamil Nadu politics.
Fact Check:
To verify the claim, relevant keyword searches were conducted online. During the research, a clarification post from Frontline magazine’s official X account, shared on April 22, 2026, was found. In the post, the publication clearly stated that the viral cover page was fake and had not been published by the magazine. It further said that the fabricated cover was being circulated online by supporters of TVK.
A review of the official Frontline website also found no edition carrying such a cover page.
Additionally, the viral image was analyzed using the AI detection tool Hive Moderation, which rated it as 92 percent likely to be AI-generated.
Conclusion:
The claim that Frontline magazine published a cover story on a “Vijay Wave” during the Tamil Nadu Assembly elections is false. The viral cover page is fake and is being circulated online to mislead people.
Executive Summary
A video showing a car caught in a web of electrical wires beneath a flyover is being widely shared on social media. In the viral video, the car appears to be suspended in mid-air, while a large number of people can be seen gathered below, watching the astonishing scene. People standing on the flyover also appear shocked as they look at the scene below. However, CyberPeace Research found that the video is not authentic but AI-generated and is being circulated with a misleading claim.
Claim:
An X user shared the viral video with the caption: “#ViralVideo: Car caught in a web of wires beneath a flyover!”
https://x.com/DBhaskarHindi/status/2107022982258553245
.
Fact Check
To verify the authenticity of the viral video showing a car caught in electrical wires beneath a flyover, we conducted a reverse image search of its keyframes using Google Lens. During the search, we found the same video uploaded on August 28, 2026, on the Instagram account of an AI content creator, @bhau_mitra_mandal_parbhani. The caption accompanying the video read: “Everyone is shocked to see this sight of a car caught in a web of electrical wires beneath a flyover! The car is dangerously stuck in the wires, drawing the attention of everyone gathered below. How did the car get here? The reason is yet to be revealed.” The caption also clearly mentioned that the visuals were AI-generated. In addition, the words “AI News” and “AI Generated Visuals” were displayed on the video itself, further indicating that the footage was artificially generated.
https://www.instagram.com/reels/DclpJHfgqXr/
To further verify the authenticity of the viral video, we analysed it using the AI detection tool Hive Detect. The tool’s analysis identified the video as AI-generated with a 99.9 per cent probability.
Another AI detection platform, WasitAI, also identified the visuals in the video as highly likely to be AI-generated.
Conclusion
The evidence gathered during our research makes it clear that the viral video showing a car caught in electrical wires beneath a flyover is not authentic but AI-generated. Our research found that the same video was originally uploaded by an AI content creator, where it was clearly labelled as AI-generated. The findings were further supported by AI detection tools, including Hive Detect and WasitAI, which also identified the visuals as highly likely to be AI-generated. Therefore, the viral video is being circulated on social media with a misleading claim, presenting AI-generated visuals as a real incident.
