London and the AI ​​Apocalypse: Real-World Risks, Threats, and City Resilience

Years ago I read author and historian Peter Ackroyd about the secret underground history of London. I became obsessed with the lost rivers, such as the Effra and the Westbourne, which still flow in darkness beneath the city. Since I started researching this piece, I’ve felt the same way about artificial intelligence, thinking of it as a largely invisible system that spreads beneath and across London, wandering through boroughs and institutions, twisting everyday life.

Across London, an AI network is already taking shape. Underneath the roads, Thames Water has more than 75,000 acoustic sensors listening for leaks (no jokes about efficiency please). In Tooting, AI listens to consultations at St George’s A&E and writes clinical notes, saving doctors an average of 47 minutes per shift; across all 32 boroughs, machine learning joins fragmented records to help London understand the journeys of people sleeping rough.

In Lewisham, AI has identified more than 3,000 overlooked sites with theoretical capacity for nearly 10,000 homes; if you hang out in Croydon, you may have been one of more than 470,000 people who had their face scanned by AI-powered cameras mounted on lampposts and compared to a police watch list. On the tube, predictive AI spots signs of trouble before a train, signal or escalator hits. Increasingly, algorithms are figuring out how groceries travel from store to doorstep. And if you drive an electric vehicle, the AI ​​can already decide when to charge your car. We are giving more of the reins to AI and the handover is picking up speed and scope.

Meanwhile, it felt a bit like the Apocalypse Olympics. Last month alone, Anthropic warned that AI models would gain “tactical intelligence targeting and conventional weapons capabilities.” OpenAI has called for international standards to govern frontier AI systems that can potentially advance faster than humans can understand, test or control them, and Bill Gates has warned that, unchecked, AI could “cause a billion deaths”. All of this reminds me of the classic Crimewatch sign: “Don’t have nightmares, sleep well!”

“The sci-fi scenario is actually not what we should be worrying about,” says Dr Stephanie Hare, co-presenter of the BBC’s Artificial Intelligence: Decoded. “The very real scenarios that really serious people are concerned about are super basic.” The biggest risk to London, she says, is good old-fashioned cyber security. Most organizations “have not kept up with their latest cyber security investments”. It’s a David-and-Goliath problem, except the advantage is reversed. “If you’re an institution or company, you have to defend your entire attack surface,” says Hare. “Where the attacker only has to get it right once.”

“Think about what that means in a city as digital as London. Imagine everything that can be hacked that you depend on,” says Hare. “Imagine their bank cards no longer work. Public transport no longer works. Imagine you want to cause maximum chaos in London. Go to a bank to prevent people from accessing their money and watch London lose it.”

Recently, the government offered the slightly surreal advice that we should all keep an emergency supply of food and water at home. France and Switzerland, says Hare, are much more prescriptive. Britain is “a bit hand-waved; like, you should have some beans. What’s wrong with us?” Being an equal opportunity disaster, I actually assumed that the much discussed “prepper pantry” was for El Niño impacts or climate change drought. But Hare’s concern is what will happen when the increasingly AI-dependent systems that get us food stop working.

London is unusually dependent on these systems. Twenty-nine percent of the 6.347 million tons of food and drink that supply the capital every year come from the city; less than one percent is produced here. City Hall described a food supply dependent on “a complex set of interdependencies and just-in-time delivery systems”. This summer, the Mayor, Lord Khan, announced that the London Resilience Unit is working on a London Food Systems Resilience Partnership, together with the food and agriculture charity Sustain to prepare for the disruption of “international shocks and crises”.

Many Londoners don’t worry about AI, or collect tins of baked beans. At Weave, a recent summit held by London technology company Loomery on “building the agent company,” there was excitement about what it calls the second wave of AI. Wave one was individual productivity: AI helps us do existing work faster. Wave two is organizational reinvention: businesses are redesigning work itself around AI agents. Remove the AI ​​and the process stops working. “Wave one was reversible. Wave two is not,” was Loomery co-founder Tim Checkley’s formulation. This is agentic AI: instead of waiting for each human request, an AI agent is given a result and proceeds with it.

David Lagnado, professor of cognitive and decision sciences at University College London, studied what happens when AI enters our decision-making processes. He is generally a fan. Recently, when he found himself waking up at 4 a.m. with a toothache, he turned to ChatGPT. “It’s completely all right. And it was quite complex,” he says. “If we use it in the right way, it could open things up for a lot of people.”

But at the moment it seems like there is a pain barrier. Recent global research on AI and jobs found that gains are skewing heavily against senior workers: employment rose 6.7 percent in senior roles but fell 2.5 percent in junior ones, exactly the jobs people need to get started.

In April, the founder of lastminute.com, former government digital czar and now peer Baroness Lane Fox was appointed by the mayor to lead a review of what AI means for London jobs. The Lane Fox review reported in July with similar findings to the global review, warning not of mass unemployment, but of a “quieter drift” towards fewer entry-level jobs, weaker career ladders and greater inequality. What if Londoners get something tangible from all this computing? Last year I found myself in an old, leaky house in South London, where a man was connecting a small data center to a hot water tank. The retired owner and his lodger had fallen into fuel poverty. The solution was apparently to move a computer to the loft. As the installation unfolded, I began to see it for what it was: a brilliant intervention.

Surrey-based Heata servers take on computing jobs normally sent to data centres, including AI workloads. Computers get hot, but data, unlike heat, is easy to move. The jobs come via the Internet; Heata pays for the electricity and the householder receives the resulting heat in their water. Heata now has servers in 100 homes, with 3,000 people waiting for one. After all the promises about how AI could make us richer and more productive, here it was delivering an advantage that you could turn on at the faucet.

Uploading, but still a foreigner. Large-scale data centers remain the preferred model for emerging AI. London has staked a significant chunk of its future on AI. More than half of Britain’s AI companies are registered here, its AI start-ups have attracted a record $3.5 billion in venture capital in 2024, and City Hall is spending millions to accelerate adoption.

Is the pursuit of AI worth the risks and distractions? If AI is about to deliver an economic miracle, London should be where we see it first. However, research shows that simply being an AI company is not a productivity windfall – the technology does not float above the real economy. Its success still depends on people, skills, capital and location.

So how do we make sure we stay in control of our city and how do we stop useful dependency turning into blind dependency? For Theo Blackwell, London’s chief digital officer, the starting point is deceptively simple: keep people in the picture. “AI can help us improve public services, but it should support people rather than replace human judgment and accountability,” he says. “If an AI system gets something wrong, there needs to be a clear way for a person to check it, challenge it and ultimately take responsibility.”

London, he says, is already trying this principle in the way the city adopts AI. City Hall’s Emerging Technology Charter sets standards for responsible use, while the London Office of Technology and Innovation brings together more than 200 people working with AI across local government.

Critics point out that London does not have a standalone AI scenario on its Risk Register, while its security measures remain organized around individual risks such as cyber attack, power outages, transport disruption, health and public disturbances. But what if AI helps a failure cascade across multiple systems at once? Who is then responsible for seeing the whole picture?

Professor Jason McEwen is interim chief scientist at the Alan Turing Institute, which has just released Frontier AI Risks: A Practical Way Forward. “We should not only think about security in the context of abstract models,” he stressed. “If we deploy these models within enterprises, in industry, as underlying workflows and processes, then we need to make sure that they are secure in those operating environments. We still obviously need to assess the security of the models themselves, but we need to do a lot more than that. It’s the security of the whole system.

In other words, it is not enough to ask whether the AI ​​itself is safe. You have to ask what we connected it to, what we allowed it to do and what happens to everything if it goes wrong.

One thing we can say with a degree of certainty is that there is no big red “off” button that solves this. “There’s been talk of kill switches lately,” says McEwen, “but once AI is deeply integrated, it’s not the case that we can always turn things off.” Imagine doing that to the electricity for a hospital, for example. McEwen says we need systems that “fall back into a graceful, safe state” to make sure the world we’re connected to AI can continue safely without it.

But a graceful decline assumes that humans still know how to take over. “The majority of people don’t do the math anymore. They just go to ChatGPT, or their favorite AI model of choice for everything, and they completely trust it,” says Hare. “Those models are often wrong. You have to know they’re wrong to challenge them. Now imagine in 30 years, a doctor, a surgeon, using AI instead of being properly trained to do these things. Imagine that across any profession. That’s actually my biggest concern. Machines are getting smarter, humans are getting dumber.

It is enough to send you to the nearest analog library. As has been said many times, reading books could be our greatest defense.