Story audio is generated using AI
The recent withdrawal of South Africa’s draft national artificial intelligence (AI) policy by the minister of the department of communications and digital technologies (DCDT), Solly Malatsi, was an extraordinary and embarrassing saga.
The document meant to govern AI had been compiled with the assistance of generative AI, and contained, on the minister’s own account, “various fictitious sources in its reference list”.
In this edition of Business Law Focus, host Evan Pickworth interviews Ashlin Perumall, a partner at Baker McKenzie in South Africa and head of the IP tech division, about what caused this strange state of affairs and what should happen next.
Listen to the conversation:
Context
A 2023 paper by Fabrizio Dell’Acqua and others in the Harvard Business School Working Paper series asked a simple question: when is AI genuinely useful to knowledge workers, and when is it actively harmful?
Their finding, in plain terms, was that AI’s capability ran along a “jagged frontier”. This is one that is sharply uneven, full of dips and peaks, and almost impossible to map from the inside. Within the frontier, AI significantly improves performance. Just outside it, the same tool degrades performance, and the user, by definition, cannot see the line.
In the interview Perumall explains that for legal work, this is a reality. Legal products are text-driven, pattern-rich and often formulaic, exactly where AI looks most capable in 2026. But it is also citation-dependent and consequence-heavy, exactly where AI’s failures are most expensive. This is because the industry’s tolerance for failure is low, and the field is generally seen as one built around exhaustive attention to detail.
“The DCDT episode is a useful illustration. Of the references in the draft policy, several pointed to academic articles and authors that simply do not exist,” he says.
“The minister’s assessment, when the withdrawal came, was direct: ‘The most plausible explanation is that AI-generated citations were included without proper verification.’ A department drafting a policy that demands human oversight of AI had itself encountered the practical difficulty of consistently applying that oversight in drafting.”
The practical consequence for law firms, in-house counsel and regulators is the same.
“Intervention cannot be discretionary or rely on individual users’ good practice. Policies alone will not achieve the rigour needed to scale the safe use of AI. It must be a verification chain: a structural process by which any AI-assisted output is checked against primary sources before it leaves the building,” says Perumall.
“Where AI genuinely helps lawyers, such as in research, factual analysis and summarising materials already in front of lawyers, the use case is real. Where it fails, in claims of fact, citations and novel legal questions, the failures are concentrated, predictable and avoidable through process.”
The DCDT episode should act as a practical reminder to the legal profession that the frontier is jagged for everyone. A government department drafting national policy can encounter the same risks as counsel before the High Court. The discipline that supports reliable output is not seniority, sophistication or good intent alone. It is verification.
Malatsi’s own framing is worth taking seriously: “This unacceptable lapse proves why vigilant human oversight over the use of artificial intelligence is critical.”
That is the right lesson. The challenge for lawyers and for the next draft of South Africa’s AI Policy is to embed that oversight in the work, rather than discover its absence afterwards.
Business Day
Would you like to comment on this article?
Sign up (it's quick and free) or sign in now.
Please read our Comment Policy before commenting.