The Black Box Paradox in AI
One of the early promises of artificial intelligence was that it could deliver the decision making free of discrimination. But the AI has impacted human lives in many aspects and very soon the humans started realizing that artificial intelligence can also suffer from the same biases as the human intelligence.
A few years ago, Amazon mostly abandoned a system it was using to screen the job applicants when it discovered it was consistently favoring men over women. Similarly, in 2019, an ostensibly race-neutral algorithm widely used hospitals and insurance companies was shown to be preferencing white people over black people for certain types of care.
When everyone is hyping that AI provides solutions to every problem, but most of these AI models operate in Black Box i.e. internal workings are a mystery to its users. Users can see the system's inputs and outputs, but they can't see what happens with in AI tool to produce those outputs and this is known as the Black Box Paradox in AI.
The Black Box Paradox refers to inherent opacity of the AI systems, where the decision-making processes are often obscure and difficult to comprehend for humans. This lack of explainability makes it challenging to understand how AI arrives at its conclusions leading to question about transparency, and reliability of the system.
Consider a Black Box model that evaluates job candidates resumes. Users can see the inputs-the resumes they feed into the AI model. And users can see the outputs-the assessments the model returns for the resumes. But users don't know exactly how the model arrives at its conclusions like what factors it considers, how it weighs those factors and so on.
Algorithm of YouTube, Facebook, Instagram can't explain why a particular video gets viral immediately after its upload. This is hidden under many layers of training of the algorithmic model of the YouTube, Facebook, and Instagram.
The Root Cause: Deep Learning
Understanding why this happens requires knowing little bit about how machine learning models are built.
Suppose you want to teach a child the difference between a Cat and a Dog. You would probably start by showing him a bunch of pictures of both Cats and Dogs, and during that process, the child would absorb some features of Cats 😺 and Dogs 🐕. Then, hopefully, when you show him a picture he never seen before, he can figure out if it's a Cat or Dog.
This method of learning by examples reveals one of the significant ways in which bias can infiltrate a machine learning model. For instance, a facial recognition algorithm is trained mostly on the images of the lighter skinned people, it may lack accuracy in identifying darker skinned individuals.
In much the same way, in real life, people are biased toward the fair skinned people considering them as more beautiful and smarter than dark skinned people because culturally people have deep learning of this thought.
Similarly, Amazon's resume screening model proved to be biased toward men because it was trained to recognize keywords from resumes of its most successful current employees - who were disproportionately men.
The deep learning algorithms are a type of machine learning algorithm that uses multilayered neural networks. Where a traditional machine learning model might use a network of one or two layers, deep learning models can have hundreds or even thousands of layers. Each layer contains multiple neurons, which are bundles of code designed to mimic the functions of the brain.
Deep neural networks can consume and analyze raw, unstructured big data sets with little human intervention. They can take in massive amounts of data, identify patterns, learn from these patterns, and use what they learn to generate new outputs, such as images, video and text.
However, these deep neural networks are inherently opaque. Users-including AI developers -can see what happens at input and output layers, also called "visible layers." They can see the data that goes in and predictions, classifications, or other content that comes out. But they do not know what happens at all network layers in between, the so-called "hidden layers."
Explainable AI (XAI) or White Box AI: Leveraging AI models while ensuring accountability and transparency.
Explainable AI (XAI) or White Box AI is a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms.
It is an emerging field that aims to make AI systems more transparent and understandable to humans. It provides the tools and techniques to explain the reasoning behind AI decisions, allowing auditors, analysts, and stakeholders to trace how these decisions are made.
By incorporating XAI, financial institutions can identify and mitigate biases, ensure compliance with regulations, build trust with customers and regulators, and unlock the full potential of AI technology.
It is crucial for an organization to have a full understanding of an AI decision-making processes with model monitoring and accountability of AI and not to trust them blindly. Explainable AI can help humans understand and explain machine learning (ML) algorithms, deep learning and neural networks.
Explainable AI Techniques
- Prediction Accuracy: Accuracy is a key component of how successful the use of AI is in everyday operations. By running simulations and comparing XAI output to the results in the training data set, the prediction accuracy can be determined. The most popular technique used for this is local interpretable Model-Agnostic Explanations (LIME), which explains the prediction of classifiers by the ML algorithm.
- Traceability: Traceability is another key technique for accomplishing XAI. This achieved, for example, by limiting the way decisions can be made and setting up a narrower scope for the ML rules and features. An example of traceability XAI technique is DeepLIFT (Deep Learning Important FeaTures), which compares the activation of each neuron to its reference neuron and shows a traceable link between each activated neuron and even shows dependencies between them.
- Decision Understanding: This is the human factor. Many people have a distrust in AI, yet work with it efficiently, they need to learn to trust it. This is accomplished by educating the team working with the AI so they can understand how and why the AI makes decisions.
Uses of Explainable AI:
- Healthcare: Accelerate diagnostic, image analysis, resource optimization and medical diagnosis. Improve transparency and traceability in decision-making for patient care. Streamline the pharmaceutical approval process with explainable AI.
- Financial Services: Improve customer experiences with a transparent loan and credit approval process. Speed credit risk, wealth management and financial crime risk assessments. Accelerate resolution of potential complaints and issues. Increase confidence in pricing, product recommendations and investment services.
- Criminal Justice: Optimize processes for prediction and risk assessment. Accelerate resolution using explainable AI on DNA analysis, prison population analysis and crime forecasting. Detect potential biases in training data and algorithms.
In conclusion, without knowing how an AI model making decisions leads to the lack of transparency and accountability in the decision making known as Black Box Paradox in AI. This generates confusion, and distrust in AI models. The problem can be addressed with the use of Explainable AI (XAI) which gives the detailed understanding of the process of decision making and leading to the better use of AI models while ensuring accountability and transparency.