Your AI isn't as smart as it sounds - Etienne Barnard — The author, Prof Etienne Barnard

In September, an Anthropic researcher resigned with a warning that the people building powerful AI “earnestly believe that it could kill us all by the end of the decade”.

Days later, King Charles, opening his AI summit in Scotland, spoke of “existential dangers” and of technologies that risk developing “darker capacities – perhaps even to take life”.

Bill Gates warned that, in the wrong hands, AI is powerful enough to drive events causing “a billion deaths”. Meanwhile, incidents made the news of rogue AI agents “breaking out” and “escaping isolation”. Much less was written about safeguards, security standards and compromised systems.

Business leaders deciding what to do with AI will get little help from either the alarmists or the enthusiasts. What they need is a more demanding standard for when to trust it.

For those of us who have worked in AI for decades, today’s systems are, in an important sense, a dream come true. What machines can now do with language once seemed out of reach.

But wonder is not certainty. That AI can do extraordinary things tells us little about what it can reliably do next, where its limits lie or whether we understand those limits well enough to trust it with particular responsibilities.

So, when an alarming headline appears, ask what was tested, under what conditions, with what access and with which safeguards. Then ask what the incident actually demonstrates. A dramatic description is not an explanation.

Extrapolating too far

The field’s history is full of predictions that machines would soon match or surpass people. Today’s systems are dramatically more capable than their predecessors, and in some respects, in software development and mathematics, for instance, they already outperform us. Progress has been exceedingly rapid. The lesson from history is that real progress tempts us to extrapolate too far.

One reason we make such confident predictions may be that AI uses language so fluently. Language is central to human experience, so we describe fluent systems in human terms: they think, they want, they worry. Those words carry assumptions about sentience and intention that the technology’s performance does not establish. A system’s behaviour deserves to be taken seriously and scrutinised closely, without letting human-like descriptions do the work of analysis.

AI’s abilities are uneven. People who work with these systems every day see them handle complex language and reasoning impressively, then fail at something a person would find trivial. In an organisation that has built AI into its workflows or customer service, such failures can be costly. They say nothing about malicious intent; what they show is that success at one task does not prove reliability at another.

That is why boards and executives should favour contained AI, built for their specific environment, over powerful but open-ended general systems. Nor should they assume that a safeguard that works in one setting will work in another.

There is a subtler risk, too: that the horse trains the rider. AI has not become conscious or developed a drive to change us, but organisations adapt to the tools they use. Leaders should ask whether their people have started trusting fluent answers without checking them, whether processes are being reorganised around what a system can produce rather than what the task requires, and whether judgment is gradually being handed over because AI seems more capable than it is.

The answer is to be precise about where and how AI is used. Before putting it to work, African businesses should answer five questions:

Name the task: What exactly is the AI being asked to do?
Test it in real-world contexts: Does the system work with actual customers? Does it understand how they use language?
Limit its exposure: What information does it really need? Does it need access to all the data you assumed it did?
Check its work: How will errors be detected? What controls are in place in high-stakes areas such as customer privacy?
Keep accountability clear: Who is responsible when the system fails? Where and when should a person step in?

Responsible AI adoption rests on a standard that is modest but demanding: use AI where it proves its value in context, constrain it in proportion to the risk, and keep responsibility with the people who chose to deploy it.