Explainable AI and the need to ensure transparency in AI models

AI is used in almost every industry whether it’s to generate insurance quotes, claims processing, loan processing, fraud detection or financial transactions, etc.

These AI applications use Machine learning models that are complex and often use sophisticated algorithms. It can be difficult for humans to understand how they make their predictions or decisions. This can make it challenging for insurers to explain their decisions to customers or regulators and can also make it difficult to identify and correct any biases or errors in the models.

This black box approach of AI can lead to a lack of trust in the decisions and predictions made by the AI model. When the inner workings of an AI model are not transparent or easily understood, it can be difficult for people to have confidence in the accuracy and fairness of the model's outputs. This can be particularly problematic in industries where the decisions made by the AI have significant consequences, such as in finance and healthcare.

In addition, lack of transparency in AI can also make it difficult to detect and correct biases or errors in the model. For example, if a model is trained on data that is not representative of the population it is intended to serve, it may make inaccurate or unfair predictions. If the inner workings of the model are not transparent, it can be difficult to identify and correct these issues.

Therefore, it's important for AI models to be as transparent as possible, and for the way they make decisions to be explainable to humans. This can help increase trust in the model and its predictions and can also help to identify and correct any biases or errors in the model.

There are currently no specific regulations that focus specifically on the transparency and fairness of machine learning models. However, there are several laws and regulations that address related issues, such as data privacy, discrimination, and consumer protection.

In the US, the Federal Trade Commission (FTC) is responsible for enforcing consumer protection laws, and has issued guidance on how companies should use AI in a way that is transparent and fair.

The General Data Protection Regulation (GDPR) in the European Union (EU) also has provisions for transparency and fairness in the use of AI. The GDPR requires companies to be transparent about their data processing activities, and to provide individuals with certain rights, such as the right to access and control their personal data.

In addition, there are some industry specific regulations, like in the financial sector, the Office of the Comptroller of the Currency (OCC) in the US has issued guidance on the use of AI in banking, which encourages banks to be transparent about their AI practices and to ensure that their models do not discriminate against certain groups of customers.

Overall, there are several laws and regulations that can be applied to ensure the transparency and fairness of AI models and that AI is used in a responsible and ethical manner.

OVERVIEW

Explainable AI (XAI) refers to the development and use of artificial intelligence (AI) systems that can provide an explanation of how they arrived at a particular decision or prediction. The goal of XAI is to create AI systems that are transparent and interpretable, so that humans can understand how the model works and why it is making certain decisions.

There are several techniques that can be used to make AI models more explainable, such as:

• Providing explanations of the input features that were most important in making a decision.
• Visualizing the model's internal workings
• Simplifying the model or using a simpler model that is easier to understand.
• Using rule-based or decision tree models that can be interpreted by humans.

It is also important to note that Explainable AI is not the same as interpretable AI, interpretable AI is the ability for human to understand the inner workings of the AI model, whereas Explainable AI is the ability for the AI model to provide an explanation of its decision to a human.

There are several techniques that are commonly used to make AI systems more explainable, including:

• Rule-based systems: These systems use a set of predefined rules to make decisions, which can be easily understood by humans.
• Decision trees: These are graphical representations of the decision-making process used by an AI model, which can help to make the model's reasoning more transparent.
• Model distillation: This technique is used to create a smaller, simpler model that mimics the behavior of a larger, more complex model, making it easier to understand and interpret.
• Attribution methods such as Local interpretable model-agnostic explanations (LIME), Shapley Additive exPlanations(SHAP), Explain Like I'm 5 (ELI5): These methods attribute the importance of features or inputs in the decision-making process of a deep learning model, it helps to understand the factors that contribute to the model's prediction.
• Counterfactual analysis: This technique is used to generate alternative scenarios that could change the outcome of the decision made by an AI model, it's help to understand the reasoning behind a decision

These are a few of the most common XAI techniques, but the field is constantly evolving and new methods are being developed. The choice of technique will depend on the specific problem and the type of model used.

PREREQUISITES

• Readers have basic knowledge of AI and ML fundamentals.
• EDA, Feature engineering, Training and Prediction/Inference is assumed to have been performed. Focus is on explainability of the model.

Using Attribution XAI techniques to understand the ML Models output

Sample Problem Statement

For the purpose of this paper, we will consider a specific business problem for which we will create a Machine Learning model and use some of the XAI techniques listed above.

The problem statement relates to understanding the risk underwriting in Health Insurance, the interplay of various attributes of the insured and see how they affect the insurance premium.

For the illustration, we will use the US health insurance dataset which is available on Kaggle. (https://www.kaggle.com/datasets/teertha/ushealthinsurancedataset).

This dataset will predict the insurance charges based on the following attributes of the insured(features):

• Age - Age of primary beneficiary
• Gender - female / male
• BMI - Body mass index, providing an understanding of body, weights that are relatively high or low relative to height, objective index of body weight (kg / m ^ 2) using the ratio of height to weight, ideally 18.5 to 24.9.
• Smoker - Smoker / Non - smoker
• Children- Number of children covered by health insurance / Number of dependents
• Region - The beneficiary's residential area in the US, northeast, southeast, southwest, northwest.

The dependent variable is Charges that are Individual medical costs billed by health insurance.

Since this is a regression problem, for this illustration we have trained the model with Random Forest Regressor algorithm. We could have used other algorithms as well, but the focus here is more on explaining the models output rather than the accuracy of the model itself.

Below mentioned are some details of the data

Credit: https://www.kaggle.com/code/teertha/us-health-insurance-eda

Sample Data

Important statistics of the data

Correlation between different attributes

XAI Attribution Techniques

Feature Importance

One of the fundamental questions we ask is what features have the most impact on predictions/inferences.

Feature importance refers to the techniques used to determine the relative importance of each feature in a machine learning model. It is a way to understand which features are most important in making accurate predictions and can be used to identify and remove redundant or unnecessary features.

In this article, we will cover two Feature importance approaches, Permutation Importance and SHAP.

Permutation Importance

Permutation importance is a method used to determine the importance of each feature in a machine learning model. It is a technique that helps to understand the relationship between features and the model's predictions. It works by measuring the change in a model's performance when the values of a single feature are randomly shuffled or permuted. The greater the change in performance, the more important that feature is.

Below is the process:

• Fit the model on the original data set
• Shuffle the values of a single feature and measure the change in the model performance
• Repeat step 2 for all features
• The feature that causes the greatest decrease in performance is considered the most important.

Permutation importance is a simple and efficient way to understand how features influence a model's predictions. It is a model-agnostic method, which means that it can be used with any type of machine learning model. It is also a robust technique, as it does not rely on any assumptions about the distribution of data or the form of the model.

It is important to note that permutation importance is a way to measure the feature importance on the model performance and not on the underlying data. It is a way to understand how the model uses the feature and not to understand the relationship between the feature and the target.

ELI5

ELI5 stands for "Explain Like I'm 5", which means to explain something in a way that is easy to understand for someone who is 5 years old. In the context of permutation importance, it means to explain how the feature importance is calculated in a simple and intuitive way.

We have used ELI5 library to show the permutation importance below:

1. Global interpretation – With weights we can look at model’s parameters and figure out at global level how the model works.

The initial figure illustrates the reduction in the model's performance as a result of random shuffling. The degree of performance change caused by shuffling a column is somewhat random. We assess the level of randomness in our calculation of permutation importance by repeating the process using multiple reshuffles. The number following the ± symbol represents the variation in performance from one reshuffle to the next.

2. Local interpretation – With explain prediction we look at a single prediction and identify features leading to that prediction.

In this example it’s the age which is the most important feature determining the charges.

Shapely Additive explanations(SHAP)

SHAP is a method for explaining the output of any machine learning model. It assigns a feature importance value for each feature for a specific prediction. The method is based on the concept of Shapley values from cooperative game theory, which allows to fairly distribute a value among a group of individuals.

The basic idea behind SHAP is that it explains the output of a model by computing the contribution of each feature to the prediction. It considers the interaction effects between features and the overall value of the model's output.

SHAP uses a specific mathematical formula to calculate the feature importance, called the Shapley value. This formula considers all possible combinations of features and assigns a value to each feature based on its contribution to the prediction.

SHAP values can be used to explain individual predictions for a specific input, by breaking down the output of the model into the contributions of each feature. The values range between the minimum and maximum possible value of the model's output and are zero for the base value (i.e., the expected output of the model without considering any feature). Positive values indicate that the feature increases the output, and negative values indicate that the feature decreases the output.

Above graph is important but it’s not too informative as it does not go beyond the importance.

Let’s look at the summary plot below generated by Shapley library. The summary plot combines feature importance with feature effects. Each point on the summary plot is a Shapley value for a feature and an instance. The position on the y-axis is determined by the feature and on the x-axis by the Shapley value. The color represents the value of the feature from low to high. Overlapping points are jittered in y-axis direction, so we get a sense of the distribution of the Shapley values per feature. The features are ordered according to their importance.

In the summary plot, we see first indications of the relationship between the value of a feature and the impact on the individual prediction.

But to see the exact form of the relationship, we must look at SHAP dependence plots. For instance, what is the distribution of effects? Is the effect of having a certain value pretty constant, or does it vary a lot depending on the values of other features. SHAP dependence contribution plots add a lot of detail.

To explain the prediction of one observation we use the force plot. For this instance, to predict the charges the features in red are shown to have a positive(increased) impact, while those in blue are shown to have a negative(reduced) impact.

Summary

In conclusion, feature importance techniques are an essential component of Explainable AI (XAI) that allow us to understand how a machine learning model makes its predictions. These methods are valuable as they provide insights into the internal workings of a model. It allows identifying errors in the model and diagnose problems, such as bias or overfitting, and allows to focus on improving the quality of the data used for the most important features to improve the model's performance. Additionally, it helps to increase trust and transparency in the model by providing explanations to the end-users. In short, feature importance techniques are a powerful tool for understanding and interpreting machine learning models which can be used to improve model performance and facilitate more effective decision making.

About Coforge

Coforge is a global digital services and solutions provider, that enables its clients to transform at the intersect of domain expertise and emerging technologies to achieve real-world business impact. We can help refine your problem statement, crystallize the benefits, and provide concrete solutions to your problems in a collaborative model.  

Authors

Siddharth Vidhani

Siddharth Vidhani is an Enterprise Architect in Digital Engineering of Coforge Technologies. He has more than 18 years of experience working in fortune 500 product companies related to Insurance, Travel, Finance and Technology. He has a strong technical leadership experience in the field of software development, IoT, and cloud architecture (AWS). He is also interested in the field of Artificial Intelligence (Machine Learning, Deep Learning - NLP, Image recognition, XAI).

Deepak Saini

Deepak Saini is AVP in CTO Organization of Coforge Technologies. He has 22+ years of IT experience in Financial Services, Telecom, Travel and Hospitality. He has strong technology leadership experience in Artificial Intelligence (Machine Learning, Deep Learning, NLP, Speech, Conversational AI, Contact Center AI, Responsible AI), Search, Intelligent Automation, Analytics, Cloud, Enterprise Metaverse Platforms, 2D/3D/AR wayfinding solutions.