Griffin on Tech: When IT goes postal and avoiding AI doing the same

Imagine being accused of theft, fired and having your career ruined because some misconfigured piece of software identified irregular financial activity in your division.

That’s the nightmare scenario that unfolded at the UK Post Office, where 736 subpostmasters and postmistresses were prosecuted for theft, fraud, and false accounting in their branches, between 2000 and 2014.

They had done nothing wrong. Instead, an accounting software system introduced by Fujitsu back in 1999 to replace manual accounting processes, produced numerous errors. In what became known as the Horizon scandal, named for the IT system Fujitsu built, Post Office staff covered up systemic problems with the accounting system, instead pinning the blame for unexpected financial shortfalls on workers responsible for financial reconciliations.

It represents one of the biggest miscarriages of justice in Britain’s history and is still playing out over 20 years later as a public inquiry this week hears more details of the lives ruined by an IT project gone rogue. Many have had their convictions overturned. But At least four of the postal workers wrongfully convicted committed suicide. Dozens more, many of them late in their careers when they were accused of fraud, were unable to find new jobs.

Butt-covering and self-denial

We’ve had our share of failed government tech projects here - the Police INCIS system and Novopay among them. But the Horizon scandal is a textbook case of not only flawed IT design and implementation, but a toxic culture of butt-covering and self-denial by management that allowed innocent people to get hurt.

Horizon serves as a sobering warning as we face the next wave of automation, driven by the generative AI systems that are set to transform virtually every industry. AI systems are already plagued with a lack of transparency and, in some cases, explainability. Now we have a rush to apply AI to large company data sets to generate insights. As Judith Collins, Nationals research, science, artificial intelligence and technology spokesperson told me this week, AI has huge potential to transform how government services are delivered.

In fact, National sees a role for AI in reviving its social investment policies formulated by Bill English when he served as finance minister in the Key government. But the risks associated with IT systems making wrong decisions that affect people’s lives are as real as ever.

MPs focus on AI

Collins has established a cross-party caucus on AI including MPs from the Green Party and Labour, with the aim of upskilling politicians on the ethical issues raised by AI. You can listen to my full interview with Collins on the new Business of Tech podcast I launched this week with my colleague Ben Moore at BusinessDesk.

Meanwhile, the Artificial Intelligence Researchers Association has released a very good discussion paper looking at the issues raised by the proliferation of generative AI services based on large language models.

"Two decades ago, artificial intelligence was merely an aspiration, but now it is an urgent reality that demands our immediate attention,” the researchers point out. They outline 12 recommendations for policy makers, including “encouraging independent certification to enhance the reliability and consistency of assessment, particularly in domains where there is a clear public interest in reliable assessment”.

The productivity promise

Judith Collins sees huge scope for AI to overhaul the experience of dealing with government. The AI researchers also see massive productivity gains from appropriate use of the technology.

“First and foremost,” they point out, “this technology can significantly boost per-capita productivity. Effects on productivity will vary by industry and role, but for some roles (e.g. copy-writing for marketing purposes) the productivity effect is likely large (in the range 30% to 300% is plausible, and 10x is not implausible for some types of work).”

If more taxpayer dollars can go towards tackling issues in healthcare, social welfare, education and justice, as a result of employing AI, that’s a great outcome for society from use if AI. But we need to lay the groundwork to get it right, put the guardrails in place, and ensure we avoid the type of scenario that has played out over 20 years in the UK and left a trail of human devastation in its wake.