Explainability Remains The Biggest Challenge To Artificial Intelligence

You might think that understanding and explaining artificial intelligence is becoming a job in itself, but it has actually become part of everyone's job. This struck me particularly hard while reading the recent report from UK Finance (and Microsoft), on the role of artificial intelligence in financial services. It shows that organisations are treating AI as a project or programme in itself, and struggling with where to pin responsibility for it, when actually their use of AI (and existing exposure to it through ad networks etc) means it's already loose in the world. That makes "explainability" - of AI itself and its outcomes - absolutely critical.

What is AI?

One first challenge is understanding what is meant by "AI" in any given context. In this report, the authors generally mean "a set of technologies that enable computers to perceive, learn, reason and assist in decision making to solve problems in ways that mimic human thinking."

We seem to have moved on from the debate about whether AI will ever move far beyond "narrow AI" (better than humans at some tasks like chess, Go or parsing vast quantities of data) to "general AI" (as good as a human mind) to superintelligence (better than humans, to the point where the machines do away with us altogether).

It seems widely accepted that we are (still) developing narrow AI and applying it to more and more data and situations, with the vague expectation (and concern) that one day it might become "general". 

The next major challenge is explaining each technology in the "set of technologies" that encompass AI. Not all are spelt out in the report, but I understand these technologies to include machine learning, neural networks, deep learning networks, natural language processing, speech recognition, image and facial recognition, speech and acoustic recognition. The report notes they are often used in conjunction  (e.g. scanning documents for hints of fraud, robotic process automation ("RPA") and personalising services for individuals or groups of customers). And it's important to understand that one or more technologies will be combined with devices or other machines in the course of biometrics, robotics and the operation and co-ordination of autonomous vehicles, aircraft, vessels and the 'Internet of things' - not ordinarily thought of in terms of financial services, but the data and decision-making in the context of these uses will be relevant for many financial institutions.

Each new report seems to bring a nugget or two of new jargon to understand, and this one alerted me to the use of "Random forests". 

What is a good use-case for AI?

The good news for the human race is that the authors recommend combining artificial and human intelligence rather than allowing the machines to work alone toward our extinction. AI can build on human intelligence by recognising patterns and anomalies in large amounts of data (think fraud detection) and can scale and automate repetitive tasks in a more predictable way to analyse and try to predict risks. The report suggests that AI Nirvana for UK financial institutions is fully automated customer on-boarding, personalised customer experience, retail advice and proactive financial management.

You might have spotted that the last two aspirations will be particularly exciting for fans of financial 'scandals'... and it's worth noting that the report on the health and motor insurance sectors added pricing, underwriting, claims handling, sales and distribution...

UK Finance rightly points out that organisations need to consider the implications of AI beyond the technical (or technological), particularly when used in the core of their businesses. Specifically, there are implications for culture, behaviour and governance from the business, social and economic perspectives. Privacy, safety, reliability, fairness (lack of bias and discrimination) are critical to safeguard, as well as adapting the workforce, communities and society for the impact on employment and skills. Again, AI can't be treated as separate or managed in a silo; and it's a challenge for all stakeholders, including regulators and governments.

Yet, while AI might be pervasive in its impact and effects, that does not mean it is ripe to be deployed in every situation (as is the case with applying process improvement methodologies like Six Sigma). The report provides some insight into identifying where AI is the right solution, as well as high-value use cases, levels of AI maturity and capabilities; and how to scale and measure returns on investment and business impact.

The Thorny Issue of Explainability...

While the UK Finance report is intended as an overview, a major criticism I have is that it only sounds a note of caution on the worrying issue of "explainability" without pointing out that explainability is not possible with technologies that have "hidden" layers of computing, such as artificial neural networks and deep learning. The report merely cautions that: 

"Where firms identify a trade-off between the level of explainability and accuracy, firms will need to consider customer outcomes carefully. Explainabilty of AI/ML is vital for customer reassurance and increasingly it is required by regulators." 

This is the point where the fans of financial scandals start stockpiling popcorn.

The relevant shortcomings and concerns associated with explainability are covered in more detail in my post on the report into the health and motor insurance sectors, including the South Square chambers report. But in summary, these mean that neural and deep learning networks, for example, are currently only really appropriate for automating decision-making where "the level of accuracy only needs to be "tolerable" for commercial parties interested only in the financial consequences... than for... issues touching on fundamental rights." 

Yet the UK Finance warning not only assumes that the use of AI and its outcomes is known by or can be explained to people within the organisation (when that may not be the case), but also assumes that organisations understand what the trade-off between explainability and accuracy means; the implications of that; and therefore whether a given use-case is actually appropriate for the application of AI technologies. A critical issue in that analysis is how to resolve any resulting disputes, whether in the courts or at the Financial Ombudsman, including identifying who is responsible where AI computing has been been outsourced and/or there are multiple external sources of data.

None of this is to say, "Stop!" (even if that were possible), but it's important to proceed with caution and for those deploying and relying on AI to be realistic in their expectations of what it can achieve and the risks it presents...

Of Caution And Realistic Expectations: AI, ANN, BDA, ML, DL, UBI, PAYD, PHYD, PAYL...

A recent report into the use of data and data analysis by the insurance industry provides some excellent insights into the pros and cons of using artificial intelligence (AI) and machine learning (ML) - or Big Data Analytics (BDA). The overall message is to proceed with caution and realistic expectations...

The report starts by contrasting in detail the old and new types of data being used by the motor and health segments in the European insurance industry: 

• Existing data sources include medical files, demographics, population data, information about the item/person insured ('exposure data') and loss data; behavioural data, frequency of hazards occuring and so on;

• New data sources include data from vehicles and other machines or devices like phones, clothing and other 'wearables' (Internet of things); social media services; call centres; location co-ordinates; genetics; and payment data.

Then the report explains the analytical tools being used, since "AI" is a term used to refer to many things (including some not mentioned in the report, like automation, robotics and autonomous vehicles). Here, we're talking algorithms, ML, artificial neural networks (ANN) and deep learning networks (DLN) - the last two being the main focus of the report.

The difference between your garden variety ANN and DLN, is the number of "hidden" layers of processing that the inputs undergo before the results pop out the other end. In a traditional computing scenario you can more readily discover that the wrong result was caused by bad data ("shit in, shit out", as the saying goes) but this may be impracticable with a single hidden layer of computing in an ANN, let alone in a DLN with its multiple hidden layers and greater "challenges in terms of accuracy, transparency, explainability and auditability of the models... which are often correlational and not causative...".

Of course, this criticism could be levelled at the human decision-making process in any major financial institution, but let's not go there...

In addition, "fair use" of algorithms relies on data that has no inherent bias. Everyone knows the story about the Amazon recruitment tool that had to be shut down because they couldn't figure out how to kill its bias against women. The challenge (I'm told) is to reintroduce randomness to data sets. Also:

As data scientists find themselves working with larger and large data sets and working harder and harder to find results that are just slightly better than random, they will also have to spend significantly more time and effort in accurately determining what exactly constitutes true randomness in the first place.

Alarmingly, the insurers are mainly using BDA tools for pricing and underwriting, claims handling, sales and distribution - so you'd think it pretty important that their processes are accurate, transparent, explainable and auditable; and that they understand what results are merely correlated as opposed to causative...

There's also a desire to use data science throughout the insurance value chain, particularly on product development using much more granular data about each potential customer (see data sources above). The Holy Grail is usage-based insurance (UBI), which could soon represent about 10% of gross premiums: 

• pay-as-you-drive (PAYD): premium based on kms driven;

• pay-how-you-drive (PHYD): premium based on driving behaviour; and

• pay-as-you-live (PAYL): premium based on lifestyle, tracking.

This can enable "micro-segmentation" - many small risk pools with more accurate risk assessments and relevant 'rating factors' for each pool - so pricing is more risk-based with less cross-subsidy from consumers who are less likely to make claims. A majority of motor insurers think the number of risk pools will increase by up to 25%, while few health insurers see that happening. 

Of course, micro-segmentation could also identify customers who insurers decide not to offer insurance (though many countries have rules requiring inclusion, or public schemes for motorists who can't otherwise get insurance, like Spain, Netherlands, Luxembourg, Belgium, Romania and Austria). Some insurers say it's just a matter of price - e.g. using telematics to allow young high risk drivers to literally 'drive down' their premiums by showing they are sensible behind the wheel. 

Increases in the number of 'rating factors' is likely to be more prevalent in the motor insurance segment, where 80% (vs 67%) are said to have a direct causal link to premium (currently driver/vehicle details, or age in health insurance), rather than indirect (such as location or affluence).

Tailoring prices ('price optimisation') has also been banned or restricted on the basis that it can be unfair - indeed the FCA has explained the factors it considers when deciding whether not price discrimination in unfair. 

Apparently 2% of firms apply BDA to the sales process, resulting in "robo-advice" (advice to customers with little or no human intervention).  BDA is also used for "chatbots" that to help customers through initial inquiries; to forcecast volumes and design loyalty programmes to retain customers; prevent fraud; to assist with post-sales assistance and complaints handling; and even to try to "introduce some demand analytics models to predict consumer behaviour into the claims settlement offer."

Key issues include how to determine when a chatbot becomes a robo-adviser; and the fact that some data is normally distributed (data about human physiology) while other data is not (human behaviour).

All of which begs the question: how you govern the use of BDA?

Naturally, firms who responded to the report claim they have no data accuracy issues and have robust governance processes in place. They don't use discriminatory variables and outputs are unbiased. But some firms say third party data is less reliable and only use it for marketing, while others outsource BDA altogether. But none of this was verified for the report, let alone whether or not outputs of ANN or DLN were 'correct' or 'accurate'.

Some firms claim they 'smoothed' the output of ML with human intervention or caps to prevent unethical outcomes.

Others were concerned that it may not be possible to meet the privacy law (GDPR) requirements to explain the means of processing or the output where ANN or DLN is used.

All of the concerns lead some expert legal commentators to suggest that ANN and DLN are more likely to be used to automate decision-making where "the level of accuracy only needs to be "tolerable" for commercial parties [who are] interested only in the financial consequences... than for individuals concerned with issues touching on fundamental rights." And there remain vast challenges in how to resolve disputes arising from the use of BDA, whether in the courts or at the Financial Ombudsman.

None of this is to say, "Stop!" But it is important to proceed with caution and for its users to be realistic in their expectations of what BDA can achieve...