Suggestions

Products & Resources

Back to all resources

Summary

Transparency is a critical step in the AI implementation process. This white paper for business and technology leaders explores real world examples of how to incorporate transparent AI into their business.

Introduction

As the technology industry continues to solidify plans to incorporate Artificial Intelligence (AI) into key business processes, pressure is increasing on CTOs and senior developers to put the right AI solutions in place. One of the most essential applications of AI is search, either on websites, apps, or more specialized search experiences.

As a CTO or senior dev, you are responsible for implementing these search experiences. When the search algorithms work, users find what they want and they’re more likely to continue using that search experience. They engage with content or purchase products and your stakeholders are satisfied with your search implementation. If users don’t find what they’re looking for easily, they’ll look elsewhere for the products or information they need. You will have lost the opportunity to direct those users, whether they be shoppers looking for a specific item, students, readers, or viewers looking for specific content, or decision makers looking for information about a product or solution. Failure to deliver the right results leads to missed opportunities for sales or engagements, slower growth for the company that relies on you, and some awkward conversations about how the AI works – or doesn't.

This is the cost of an AI search algorithm that isn’t optimized, and a large portion of suboptimal AI experiences are the result of not knowing how or why an AI solution is failing. In short, these solutions are not transparent with their decision making processes during implementation or fine-tuning, and, as a result, they lose the trust of both their end-users and the stakeholders that rely upon them.

Transparency is a critical step in the AI implementation process. Without it, any attempt to optimize the results would be nothing more than blindly adjusting training data and model parameters until the model, hopefully, gets better. Even then, without understanding why the model got better from those adjustments, there are no guarantees that future data points (that the new and improved model was not trained or tested on) will result in the same improved performance. It only takes a handful of bad experiences to start losing the trust of end-users – a risk you don’t want to take with your search algorithm.

The dependance of easy optimization, and eventually, user trust, on AI transparency is the reason it is one of the fastest growing sub-fields of Artificial Intelligence. But what does AI transparency actually mean, in a practical sense? Zendesk defines it as “understanding how artificial intelligence systems make decisions, why they produce specific results, and what data they’re using.”

AI transparency, like many umbrella terms, is both helpful when referring to a class of AI challenges, and also not specific enough to begin solving them. But AI transparency includes ideas like AI interpretability and AI explainability, and this is where we can start being prescriptive about implementing transparent AI solutions. Transparency, explainability, and interpretability are often used interchangeably, but for a practitioner of these concepts, it bears being specific.

According to Richard Mann’s Unlocking AI Algorithms: A Journey into Explainable Artificial Intelligence, “Explainable AI (XAI) is a set of principles, techniques, and procedures used to help the users and creators of AI models understand how these models make decisions.” This is a good example because it’s simple and human-centric, but it also sets the stage for treating explainability as a tool for improving the AI itself, the problems it’s meant to solve, and the business outcomes for which it’s responsible.

Interpretable AI, though similar to transparency, is a separate concept from explainable AI. IBM defines interpretability as "the degree to which an observer can understand the cause of a decision. It is the success rate that humans can predict for the result of an AI output, while explainability goes a step further and looks at how the AI arrived at the result." By these definitions, interpretability is an ante-hoc understanding of a model's decision making process, or the factors that will impact decisions, while explainability is a post-hoc understanding or analysis of exactly how a model arrived at a decision.

See the glossary of terms at the end for easy definitions of these concepts as they relate to AI transparency, or see this article for a more conceptual overview of explainable AI principles

With these definitions in mind, let's start exploring some real world examples of how to incorporate transparent AI into your business.

Simpler self-interpretable models

Some models are simple enough to be self-interpretable. According to Built In, “self-interpretable models are, themselves, the explanations, and can be directly read and interpreted by a human... Some of the most common self-interpretable models include decision trees and regression models, including logistic regression.”

Although it uses interpretable and explainable interchangeably, and one can argue that the models listed are both self-interpretable and self-explanatory, this is a good place to start understanding how to make your models trustworthy and transparent.

If you’re an experienced hands-on-keys data scientist, and you already know how best to implement and explain the aforementioned models, you can skip ahead to p.13, Model-agnostic transparency methods. But if you’re a CTO, business leader, or someone who works alongside data science, read on to learn how simple self-interpretable models can incorporate AI transparency best practices.

Linear regression

Linear regression is a type of machine learning model that creates a linear relationship between two variables in order to predict future data points along the same line. It is a common way of using known metrics, like searches, clicks, or page views to fill in or forecast metrics that are harder to track or haven’t occurred yet, like checkouts, sales, or other downstream KPIs with some but limited availability. Linear regression models are also considered self-interpretable and self-explanatory, which is good because if you don’t trust the model that generated missing or future data, you cannot trust the data itself, and it will experience low adoption across the organization.

Pushing “go” on a linear regression’s fit method doesn’t automatically spit out an explanation. Nor can you just eyeball its visualized output (a line of best fit, as shown below) and prove that it's actually the line of best fit, so linear regression models are not self-explanatory in the way some human decisions are, or in the way that a simpler technology, like tag-based click tracking is.

Even so, linear regression is one of the most explainable and interpretable forms of AI, because you can actually do the math for several data points and see how the line of best fit was chosen. A linear regression model minimizes the below loss function by making sure the squared average of the difference between each data point and the chosen line is as small as possible.

S = Σ(i=1..n) d_i²
S = Σ(i=1..n) [y_i − f(x_i)]²
S = d_1² + d_2² + d_3² + … + d_n²

Linear regression models, like most of self-interpretable models, are very simple code, making it easy to trust that they were implemented correctly. Here’s a whole machine learning model in python, given that you’ve already defined your columns as dependent variable y and independent variables x.

from sklearn.linear_model import LinearRegression
lrm = LinearRegression()
lrm.fit(x_train, y_train)
predictions = lrm.predict(x_test)

This usually requires data cleansing up top, plus splitting your data between training and testing data sets, and then evaluating your model below. It's simple code doing (lots of) simple math that is easy to recreate.

Linear regression is also unique in the sense that you can explain the algorithms with one metric from middle school, the slope of a line, or the linear regression model’s coefficient. Here is the single line of code:

lrm.coef_

This outputs a breakdown of how much dependent variable, x, will increase for a one unit increase in an independent variable, y. If your linear regression model is predicting the number of searches performed on your site search experience from the number of views on your website, a coefficient of two can be explained as: “for each additional view to the site, expect two additional searches.”

This is an example of a very simple ante-hoc interpretability metric, which lets you know how the model will predict future data points, as well as a post-hoc explainability metric that allows you to explain a given data point by placing it on the line of best fit. You can even predict future data points with the given information, like the example above.

But no model is perfect. The trust issue is still apparent if the model starts missing predictions and incorrect predictions might be difficult to explain if you believe the model to be infallible. A good data scientist can get you metrics quickly that prove how much we should trust this model (more on that in the next section), making it very easily explainable, even when it is wrong some small percentage of the time.

Root Mean Squared Error (RMSE) is the measure for evaluating how “off” a linear model’s predictions are from the real checkable test data points. It is an important metric for interpretability, as well as explainability, because you can understand how well that model should perform before actually deploying it.

Logistic regression and other classification models

Let’s zoom in on how model performance relates to interpretability and explainability. To discuss model performance, we need to circle back on a few lines of code not addressed above: the train-test split. While it's part of any machine learning script, it's not actually part of the model itself. The simplest script to put a machine learning model in place would follow this format:

  1. Cleanse and collect the data, which is where data scientists spend most of their time
  2. Import a model, similar to the linear regression above, but could be different for more complex models or in-house algorithms
  3. Split data into (1) predictors and dependent variable, and then further into (2) training and testing data, which we’ll talk about right now
  4. Instantiate the model (with customer parameters if necessary)
  5. Fit the model (train it on the training data set)
  6. Make predictions
  7. Analyze the predictions (against the testing data set)

Here’s the python code that comprises that train-test split (step 3.2), on the famous titanic survivor dataset, predicting whether or not a passenger survived based on a variety of factors, including gender, class, age, etc.

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(train.drop('Survived',axis=1),
train['Survived'], test_size=0.30, random_state=101)

Why do we need to do this split? Because net new predictions cannot be easily checked, splitting the data into training and testing data sets allows you to train the model on one data set and then check the models predictions against the testing data set, kind of like a teacher only putting questions on a test that they already have an answer key for, instead of testing students on problems no one has seen before. It does us no good to say “I used all the data and the model is trained, I’m ready for it to churn out predictions,” because the immediate next question is “how good are those predictions?” This is why we hold back some data as testing data. We train the model on x_train and y_train, then we use x_test to predict, then we check those predictions against y_test. Only when we’re comfortable with performance do we predict net new data points that we cannot check, knowing that new predictions should be, say, 98% correct.

As complexity increases, transparency inherently decreases.

This process of checking if a model is learning well and performing well enough before deploying it to new data is called supervised learning. And supervised learning makes for very easy interpretations and explanation.

Again, if you’re a data scientist, this might be obvious to you, but if you’re a CTO or data science-adjacent, you might wonder how to know how accurate models are, and to what degree.

A confusion matrix, a common visual analysis of a model’s performance, shows how a classification model like logistic regression got things right or wrong:

  Actual: Positive (1) Actual: Negative (0)
Predicted: Positive (1) TP FP
Predicted: Negative (0) FN TN

True Positives (TP) and True Negatives (TN) are simple to define: did your model predict things correctly for the positive and negative outcomes, respectively. If you’re predicting when users will click on a button, did you correctly predict when they would click (true positive) and when they wouldn’t (true negative)? False Positives (FP) are when your model thought they would click but they did not (you predicted a positive outcome but that was wrong). False Negatives (FN) are when you thought they would not click but they did (you predicted a negative outcome but that prediction was false).

Again, outputting a confusion matrix like the image above is simple code but if you plan to incorporate it into a model and explain the model’s reliability, knowing how and when to use it as well as what it means is part of being transparent:

from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test,pred))

Reductionism alert! This is basically what self driving cars do. Sure there are complex neural networks in place to perform the task, not just a simple logistic regression, but the goal is the same: to classify things.

They classify sections of an image as either “car” or “not car.” Your car will want to avoid cars. Your car will try to further classify “not car” as something else, possibly passing it to a different machine learning model or group of neurons in a network that specialize in classifying objects as people, signposts, or wisps of fog. (Note: this is why more complex AI is referred to as an “algorithm” and a single machine learning instance or script is simply a “model”).

The price of classifying a “car” as “not car” (a false negative) could be the life of the passengers. Classifying a deformed patch of concrete as a car might cause a hard break or unnecessary swerve.

To have transparent, trustworthy AI, your data science teams not only need to understand what types of errors you have, they also need to optimize for the least harmful ones (which means they need to be domain experts as well as data scientists). Then they need to optimize for model performance as a whole (which means your data scientists need to be data scientists as well as domain experts).

Precision, recall, and f1 scores compare predictions from a classification model like logistic regression or k-nearest neighbors to test values, so that models can be deemed accurate and trustworthy when deployed, or inaccurate so you can collect more data, choose a new model, or adjust parameters to improve accuracy.

Here’s an example of a classification report for a logistic regression model on that same Titanic dataset:

from sklearn.metrics import classification_report
print(classification_report(y_test,predictions))
             precision    recall  f1-score   support

          0       0.81      0.93      0.86       163
          1       0.85      0.65      0.74       104

avg/total         0.82      0.82      0.81       267

This train-test split and then the eventual model analysis are fundamental to:

  1. Picking the right model that performs well
  2. Making sure that the model is trained on enough data to perform its task
  3. Knowing to what degree we can trust our model’s predictions
  4. Adjusting for the contextual impact of certain types of errors like false positives or false negatives

All of this is to say that transparent AI needs more than just the right models in place, it needs the right teams and the right business processes to support it, including:

  1. Good data for training
  2. Good data teams to implement the right models or algorithms
  3. Thorough and continuous analysis to make sure the model is achieving its stated objective

All that and these models were supposed to be self-interpretable!

Complex self-interpretable models

Linear regression, logistic regression, and other simple classification models like k-nearest neighbors are pretty easy to visualize, understand, and calculate, thus have simpler explainability and interpretability methods. But here’s where we turn the corner.

Decision trees

Decision trees are a machine learning model adept at both classification and regression problems. They can handle lots of information at once. Because they follow the same basic flow and roughly the same code as the above machine learning scripts, they’re extremely powerful tools for solving problems using AI. With more complex problems, you need more powerful models, but the trade-off is they also become less transparent. This complexity-transparency trade-off will become apparent as we step through the difference between decision trees and the more complex random forest model.

Decision trees use mathematical formulae to maximize information gain with each logical split, which is to say they try to make splits that result in the greatest number of one class’ outcomes being put on one side. For example, if most of the time when it's not raining you play golf, then that's where the decision tree will make the split. Node one on the decision tree will be “is raining” and that first split will be “true” or “false.” Node two might be something like “temperature” and that split will be “>70” or “<70,” if you often don’t play when it's above that.

If this seems like an oversimplification or you're lost, don’t worry, the point is that decision trees are easily explainable, because you can see the nodes, you can identify the criteria for a split, and you can follow a decision down the tree’s branches.

Here’s an example from the healthcare industry that predicts cancer survivability instead of golf. And instead of temperature and weather, the patient data used is “Age,” “Operation Date,” and number of “Nodes.” A few lines of python code in a jupyter notebook and you’ll print out something like this:

It’s immediately apparent that when dealing with numerical data on a decision tree, there are lots more splits than just “down the middle,” and there are a lot more splits than just across the number of variables. But it is otherwise pretty easy to follow the tree to its “leaf” for a specific instance, so you can figure out how a decision tree made its prediction. As long as you have the code to spit out a visualization of the tree, which this Medium article walks us through, it is very explainable AI.

Random forests

Now how about this? Still easy to follow?

Aside from the fact that there are a lot more trees (and the image is necessarily much smaller to fit them), random forests “shuffle the deck” in terms of where they choose their splits. They’re no longer just maximizing the information gain with each split, each tree is choosing different splits to eventually account for future data. By avoiding overfitting, they often succeed where single trees fail. Also, rather than just one tree, or the five shown above, they’re usually simulating hundreds of estimators.

This is like following one user's journey through a search engine and then getting a map of all user journeys. It's another layer of extrapolation as you “zoom out” but given the understanding and trust in the underlying concept it lends itself well to understanding and trusting the concept at scale.

You can’t necessarily follow a single tree and say “this is how the model made its prediction” since the random forest model is aggregating the individual decisions of multiple trees into one final decision, either by voting, averaging, or outputting a percentage chance of an outcome. But the decision is explainable in simple-ish terms.

You can still begin to explain decision tree model accuracy with confusion matrices to see false positives and false negatives, and weigh the impact of various types of errors in the real world. You can still use classification reports to evaluate model performance and fine-tune if needed. These things are foundational to good data science and thus AI explainability, as we’ve discussed in the Logistic Regression section of this article.

But there’s one more thing that decision trees and random forests add to our explainability tool kit: an importance score.

The importance score is our first example of a true post-hoc AI explainability metric that explains how a model makes its decisions after the fact, without necessarily analyzing how the model works or dissecting its mechanisms. Importance scores tend to be global methods of interpretation that give insight into how the model makes its decisions for all of the data points, rather than backtracking to explain a single decision, as local explainability methods do.

Importance scores are easy and fundamental for explaining random forests’ decisions (see article for more info).

Simple algorithms

Right now we’ll quickly accelerate from stepping through individual types of machine learning models to focusing on how AI algorithms in general lose transparency with increased complexity. Then, we’ll begin to talk about some explainability methods that can be used for any type of machine learning model.

Understanding the extra layer of extrapolation – where we go from “models” to “algorithms” – is both important to AI explainability and to data science teams that want to implement the most effective models.

An algorithm is loosely defined as a set of steps for a program to follow. Technically, any script that incorporates just one machine learning model can fairly be called an algorithm by the formal definition. After all, there are also multiple steps of data cleansing, splitting, and outputting various metrics.

But as hinted at in the self-driving car example, when information starts hitting multiple models, you shift from single models to algorithms of models.

Grid Search

GridSearch is a method that systematically tests multiple models to use the one that performs best.

Many machine learning models have parameters that are passed into the model. Most of these parameters come with default values that are rarely adjusted. Some examples of these criteria, shown below in the standard fit method for their respective scikit-learn models, include number of clusters for k-means clustering, number of estimators for random forests, and C or gamma values for support vector classifiers that calibrate bias/variance to adjust for under/overfitting. Note: these are three separate examples of how to instantiate different models with different non-default parameters, not sequential lines of code, and in the above examples we left some of these fields blank, which, in python, allows the function to use the default parameters.

kmeans = KMeans(n_clusters=4)
rfc = RandomForestClassifier(n_estimators=600)
svc_model = SVC(C=10, gamma=0.1)

GridSearch will take a criteria-less machine learning model as an input and run it multiple times with different criteria to maximize an accuracy score. It's like giving multiple interviewee’s the same real-world problem in an interview, hiring the person that does best, and saying, “congratulations, you’re running this project now, because your model performed so well in the interview process!”

Specifically for the Support Vector Classifier (SVC) model, another common machine learning model that’s a bit more complex than the ones discussed above, you can pass different C and gamma values to allow Grid Search to select the most accurate one.

from sklearn.model_selection import GridSearchCV
param_grid = {'C': [0.1,1, 10, 100], 'gamma': [1,0.1,0.01,0.001]}
grid = GridSearchCV(SVC(), param_grid)
grid.fit(X_train,y_train)

Note how the SVC model is instantiated within the GridSearchCV function with no set parameters, and that the GridSearch algorithm itself is now the object to call the fit method on, signaling that GridSearchCV is an algorithm that incorporates a machine learning model and the dictionary of parameters passed to it.

GridSearch doesn’t necessarily reduce AI transparency. It's all about putting the right model in place whereby all the normal methods of explainability and interpretability above still hold true. It's an aside to make sure that the path from data to prediction is the best, straightest line possible.

Gradient boosting

Another way we go from single models to algorithms is gradient boosting, where models are strung together so that each can focus on correcting the mistakes of their predecessors — not unlike humans learning from their parents. XGBoost is one such python library for gradient boosted decision trees and can be instantiated in much the same way as scikit-learn has in the above code examples. You can see an example of XGBoost in action in the XAI-for-practitioners github repository.

Gradient boosting algorithms increase the complexity of your algorithm and can make a self-interpretable model like a decision tree only explainable after the fact — similar to a random forest classifier increasing model complexity beyond visual interpretability. Remember the picture of one decision tree vs. a forest? Rather than side-by-side, imagine them feeding into one another. Remember: as complexity increases, transparency inherently decreases.

Supervised learning makes for very easy interpretations and explanations.

So post-hoc explainability methodologies become extremely important on “boosted” algorithms and other types of AI algorithms, and eventually we’re left with even fewer options for transparency methods on deep learning and neural networks.

Model-agnostic transparency methods

Because many of these models have become so complex, there are no longer specific ways to dig into each model, like in the linear regression coefficient example. As such, AI transparency becomes a field in and of itself, putting algorithms on top of many of these “black-box” models. As the field of AI explainability advances, whole classes of explainability and interpretability metrics are developed that can apply to almost any model. These can roughly be broken down into two types: global and local.

Global model agnostic explainability methods are their own algorithms that sit on top of the AI algorithm and give a comprehensive explanation of the entire data set, while local model agnostic explainability methods aim to explain individual predictions.

These model agnostic explainability methods are algorithms themselves, built to analyze the AI algorithm in question, post-hoc, similar to a SaaS company having an analytics product that accompanies their core products. Just as certain products increase the value, adoption, or usability of other products, these algorithms increase the value, adoption, and performance of your AI algorithms.

Global model-agnostic methods

Global methods excel at showing how important specific features are to the models at large. Consider this example: your company is running a random forest classifier to try to predict if the customers that subscribe to one of your services are at risk of churning. You import all of your salesforce fields for each subscriber (past and present) and train the model on all of these fields. Great, you now have an “at-risk” list to focus your attention on, based on the subscribers that churned in the past.

But why did those subscribers churn, and how do you provide better service in the future? Understanding how important each field is to that random forest classifier can help you know what could prevent those businesses from canceling their subscription! Are you charging fees that you need to reduce? Is the adoption of certain feature sets essential for your subscribers to see value? Is your service model making it easy for them to get in touch with you and work through issues? If your model is trained on metrics like number of account managers assigned in the last three years, features adopted, or cost, you take one step closer to knowing. The transparency method might even provide more value than the model itself.

Permutation feature importance (the official name for importance score, as discussed in the random forests section above) is one global method that measures the increase in prediction error when each feature is altered or removed from the training set, thereby calculating a score for how important each feature is to the prediction’s accuracy (see Importance Score chart on page 11, for an example).1

Partial Dependence Plots (PDPs) map increases in each feature to marginal increases in predicted value (for regression problems) or average increase to target class probability (for classification problems). Similar to how the regression coefficient shows how much of an increase to expect for the dependent variable when independent variables are increased, PDPs work even when the relationship between the variables is non-linear, unlike the coef_ method on the linear regression model.

1 Reference.

Accumulated Local Effect (ALE) plots calculate differences in predictions for small windows of the feature values. ALE plots help data scientists visualize how extremes in the data impact the model’s decision making. Generally speaking, if there’s an extreme difference for one feature, then that feature is important to the model’s decision making process. If there’s not much difference to the model’s outcome at a feature’s maximum and minimum values, then that feature is not very important. We see that (for an airline passenger customer satisfaction data set) only extreme values of inflight wifi service have a significant impact on model predictions, while personal travel flights are generally more likely to impact customer satisfaction negatively, across the board. Ease of online booking seems polarizing but the scale is very small, and no adjustments to that metric have any significant impact on the model’s prediction of customer satisfaction (making it a candidate for removal from the dataset).

Local model-agnostic methods

Local methods try to explain individual predictions. In the example where you were trying to retain customers for one of your company’s subscription services, perhaps you got your random forest model in place and it predicted the churn of a key subscriber. That subscriber “went dark” and then did indeed churn, with no explanation as to why. Local model-agnostic transparency methods will let you dig into the model’s prediction and have it explain why it classified that client as “at-risk.” Perhaps you may never know the exact reason they churned, but your model can at least give you a solid guess, and in the future you can attempt to correct these issues for similar users that are “at-risk” but have not yet churned.

Individual Conditional Expectation (ICE) is similar to a PDP. While a PDP shows an average change to prediction as a feature changes across your entire data set, ICE shows the prediction dependence for each instance. It’s usually an overwhelming amount of data to visualize and digest if you’re looking to isolate a single prediction, but it can help data scientists identify bands of feature-prediction interaction, rather than seeing the aggregated PDP plot. It’s similar to why a data analyst would look at a box plot, or quartiles, rather than just a simple average of a metric.

Here’s another example of the airline satisfaction example, where you can again see the band of how polarizing inflight wifi service is for the many individual outcomes of the customer satisfaction prediction.

A SHAPely additive explanation, SHAP for short, is an approach stolen from cooperative game theory that lets you drill down to specific examples by treating each feature of the dataset as a player in a game (the model), looking to either take part in a game or not take part in a game based on their gain or loss from the communal gain or prediction value. A fraction of the PDP, rather than a datapoint that gets averaged into the PDP, SHAP allows you to see a contribution to model output for each feature and each data point. The below image shows how much each feature in a dataset (in this case housing data) contributes to going from the base value of the dataset, in general, to the specific prediction in question (to predict the natural log of the median home price):

A Local Interpretable Model-agnostic Explanation, or LIME, is another common post-hoc explanation for single data points achieved by sampling data points around the targeted instance, and weighing the decisions for those other data points based on how close they are to the target data point in space. The output shows the most important features to the model’s decision, the values for that specific data point for those five features, and the probability of outcome (if the original model is performing a classification task), in this case passenger survivability aboard the Titanic:

exp.show_in_notebook(show_table=True)

NLP, Neural Networks, Deep Learning, and LLMs

Here, let’s discuss a few concepts to help understand how neural networks, specifically Transformer-based (The T in Chat-GPT) models focusing on natural language processing (NLP) relate to transparent, trustworthy AI.

Let's start with the basics of NLP. An early step in any NLP algorithm is reducing a word to a numerical code, called tokenization, and associating that word with other words in various ways. Compressing the meaning and associations of that word into a vector in space is a process called vectorization.

Julien Lemoine, co-founder and former CTO at Algolia explains more about the process of tokenization and the next steps in NLP in this article.

Vectorization and dimensionality reduction use math that’s hard to unpack. This is relevant to AI transparency because the vectors that express a single word meaning in most language models are immediately unreadable to humans. Just picture that each dot below is a lot more than a dot, it's a semantic understanding of a word, and all of its associated words in the training set.

Complex AI algorithms aren’t very easy to deconstruct and hold the parts in your hands. So we develop a series of other algorithms or methodologies to analyze how the model comes to a specific decision (using local methods like ICE, SHAP, and LIME) or makes decisions in general (using global methods PDP, ALE, etc.).

After all, even most humans cannot express the full meaning of a word succinctly. You could probably give a solid definition of almost any word in your language even without a dictionary, but can you really encapsulate everything the word “home” means to you?

Neural networks, specifically ones that work with natural language, are a bit more specialized. Being on the forefront of technology, many of them have their own sets of tools for visualizing, analyzing, or otherwise attempting to explain the decisions of an internal mechanism.

For instance, self-attention models like Bidirectional Encoder Representations from Transformers (BERT) have some incredible tools that allow developers to visualize word associations. You can think of self-attention models like BERT as a set of super word associations, so the model not only understands what an individual word means in general, but understands its context within a sentence or text input.

Here’s an example of a visualization for a self-attention mechanism.

There are some obvious drawbacks. If you dig into Vig’s github repository, you may notice that there are “layer” and “head” dropdowns with many options when looking at a single visualization. This is because this language model is a neural network, where a lot of different inputs are being passed around within a complex AI algorithm, not just from neuron to neuron, but to multiple heads that put their learnings together (not like the way a random forest looks for agreement from multiple tree based models by averaging or voting; more like the way you have multiple organizations within a corporation that specialize in different parts of the company’s functioning, but all seek to make the company successful).

If there are 12 layers and 12 heads, the algorithm is performing 144 analyses of language at a time that seek to understand the ways the words in an input are associated with each other.

Each of the 144 analyses might be:

  • What is the word before the word in question
  • What is the word afterward
  • What is the object of this noun
  • What noun does this pronoun refer to

But these are not grammar-based learnings, nor human supervised; this is the model learning language in a similar way to humans — paying attention to context. And then doing that lot.

On top of that complexity, within each layer-head combination, you have to scroll down each word to see how it relates to each other word. It’s impossible to succinctly display one layer-head combination in an image without using an animated gif or video of scrolling through each word. So the actual number of calculations going on in this BERT algorithm are closer to: 144 times the number of words or special characters in the input multiplied by the number of words or special characters in the input. A best guess is that there are about 144*11² calculations, or roughly 17,000 — but you’d have to look through a lot of visualizations to be sure.

Other neural networks or deep learning algorithms have similar visualizations. TensorFlow has this visualization for a deep learning neural network. You can see from the below image, that despite the complexity in reading and analyzing the algorithm’s decision making process from this image in its current state, it's still hiding three layers of processing. These types of visualizations do a better job of showing how your model’s accuracy will react to adjustments in neuron or layer counts, rather than actually helping you understand how decisions are made.

So there’s so much going on in a neural network, that it's hard for a human to distill understanding from so much information. While these visualizations allow you to take apart the pieces of an algorithm and look at them, they still can’t tell you everything about how decisions are made. Returning to the example of putting a search experience in place, if any of the above models represented your core search algorithm, would you have any idea how to improve it to make up for the searches that don’t return relevant results? One would hope that there are better ways to explain something like a search engine algorithm’s output for a specific search if user trust and revenues depend upon the result.

Another drawback to the above visualizations, specifically in the case of BERT, is how computationally taxing these visualizations are. In fact, the owner of the self-attention github repository notes: “This tool is designed for shorter inputs and may run slowly if the input text is very long and/or the model is very large,” and then goes on to detail ways to pare things down. If you’re a CTO or architect, you might be thinking: “computationally taxing = expensive.”

So, like trying to explain the word “home”, understanding everything a BERT model knows about a word or phrase is very difficult and labor intensive.

But why is it so difficult? Because a BERT model’s job is to understand all of language. In earlier examples, you were training a model to predict a y value based on some x values, but Large Language Models (LLM’s) don’t really have one specific task, other than to learn language in general, as a foundation for any subsequent tasks you might give it. That’s why these types of models are called “foundational” or “pre-trained” (the P in Chat-GPT).

In order to trust AI solutions, oversight needs to be built into the solution, and possibly even into the way the algorithm itself works.

Purpose Built AI Solutions

From discussing individual machine learning models to talking about AI algorithms, let’s consider a complete AI “solution.” An AI solution isn't focused on the model or algorithm, but how that model or algorithm can work within a functional or business context. So for the purposes of AI explainability, foundational models, popularized via high profile generative AI (the G in Chat-GPT), might not be the right answer.

For starters they’re trained on just about the whole internet. AI industry veteran and current MIT Research Affiliate Irving Wladawsky-Berger brings up this issue in his blog post, The Emerging Data-Centric AI Era: Where Will All the Data Come From? He recently moderated a panel that discusses exactly this topic, at the 2024 MIT Sloan CIO Symposium. “Generative AI models are highly dependent on the quality and quantity of data used in their training. But, many of these models are being trained on vast, diverse, and inconsistently documented datasets that have been raising serious concerns about the legal and ethical risks involved,” the panel brief notes. “The panel will discuss the complexities of using large, diverse datasets in the training of generative AI models, the difficulties involved in understanding the provenance of the data being used, and the kinds of tools and standards needed for the responsible use of these powerful models.”

Further, since foundational models don’t have the purpose in mind when they’re trained they might not perform as well. In that same panel, Mike Mason, chief AI officer of the technology consultancy Thoughtworks, said, “We did some work with a [Deep Learning]-based application for a fashion and apparel retailer that recommends articles as clients are browsing their online catalog, and their existing system was so well trained with many years worth of data, and its algorithms were so highly tuned that the new GenAI model actually performed worse. Depending on what you’re trying to accomplish, you may be better off with a highly tuned domain-based [Deep Learning] model than with a very general model.” Wladawsky-Berger summarizes the conversation by writing, “there are instances where a relatively small, well trained, task-specific model gives better results than a large, general purpose GenAI model.”

Lemoine helps form a causal pathway from highly tuned, domain-based models to strong performance. He says, “If a company cannot tell you why [search] results are ranked in a particular order, it will be impossible for you to test alternative ways to tune and improve the results.” Hence, transparent AI is very closely linked with AI that performs well.

Lastly, AI transparency requires oversight. In order to trust AI solutions, oversight needs to be built into the solution, and possibly even into the way the algorithm itself works.

Algolia provides one example on how human oversight can ensure a model’s output is correct: by allowing customers of their AI Search solution to adjust rankings of search results manually, as well as providing insight into why a certain output was generated by a search. See the below image of a ranking criteria report:

Ranking criterion Value
0 Sort-by: featured false
1 Sort-by: hasImg true
2 Number of typos 0
3 Matching words 1
4 Matching filters 0
5 Proximity distance 0
6 Best matching attribute title (1st word)
7 Exact words count 0
8 Custom Ranking salesRank = 1
9 Personalization Score 0

Notice how in the ranking criteria, whether or not a customer chose to “feature” a result is ranking criteria zero (even before ranking criteria one in the hierarchy) and the custom ranking criterion also weighs in. This report is not unlike the LIME method, but rather than showing the top five features that contributed, it shows all possible features.

And like the SHAP method discussed early, it shows whether or not each feature (or criterion) impacted the results, sometimes in a binary/boolean fashion, like true or false, and other times in a numerical fashion, like the number of matching words, the salesRank for the Custom Ranking criterion, or the personalization score. Consider incorporating something similar into your own transparent AI models, that borrow from several well-established AI transparency methods.

Conclusion

Let’s tie it all back together, to make a playbook for transparent AI, start to finish.

  • As Wladawsky-Berger notes, transparent AI starts with good data
  • Foundational models can offer a head start to training, but that comes at the cost of transparency, since you don’t always know what they’re trained on
  • Consider purpose-built AI, trained on specific data-sets relevant to its task if your business requires a more focused solution

Next, like all other AI, transparent AI needs good data teams and good model selection, in order to understand:

  • Error rates
  • The impacts of certain types of errors
  • How best to mitigate the most harmful ones, either by adjusting certain model parameters, or by using a more complex algorithm to put the best solution in place.

After we know that models are performing well, and their potential impacts are as positive as possible, it's time to start interpreting how models come to their decisions. Interpretability methods become extremely important as models are being built out and eventually rolled out. As both Mason and Lemoine noted in separate articles, these transparency methods are extremely important for fine-tuning models as they’re being tested on more and more data. Explainability methods allow for continued analysis of new decisions, and you should consider how your models can most efficiently continue to report on their decisions, without requiring endless python scripts, using up costly data scientist hours, or purchasing GPUs from your cloud provider.

Hopefully, with these steps in mind, you’re ready to start considering transparent AI solutions for your search experience implementation. If you are, in fact, the CTO or senior dev responsible for implementing site search for your business, you may be looking for a complete AI solution that is not only transparent and fine-tune-able, but lightning fast and highly customizable.

Algolia users don’t have to rely on an algorithm’s interpretability techniques or worry about model explainability. Our search-providing customers enjoy a white-box model of transparency for how search relevance is computed. For example, our users can see how search results are ranked based on personalization and relevance factors, then manually adjust according to their needs.

About Algolia

Algolia is the world's only end-to-end AI search and discovery platform. Our engineers invented a breakthrough use of AI to create exponentially better search & discovery. Algolia's proprietary NeuralSearch tech combines vector-based natural language processing & keyword matching in a single API.

Algolia powers 1.75 trillion search requests a year or more than 30 billion a week enabling more than 17,000 customers in 150+ countries to build blazing-fast and relevant search and discovery experiences for their in-app users and/or online visitors using any web, mobile or voice device – by surfacing the desired content instantly and at scale.

Request a demo of Algolia’s AI Search solution at algolia.com/demorequest or read more at algolia.com/blog/ai/

Appendix A: Glossary of Terms

Term Meaning/description (as it relates to AI transparency)
AI Explainability Seeing how the AI algorithm came to its decision, post hoc
AI Interpretability Predicting what an AI algorithm might come up with based on what you know about its inputs, or deducing what information it will use to make decisions, generally, ante hoc (prior to the model running)
AI Transparency An umbrella term including AI explainability, AI transparency, governance, and oversight, among other things
Algorithm A complex set of rules that determine how information moves from model to model or function to function within an artificial intelligence program
Attention A mechanism commonly used in natural language processing neural networks that understand associations between words, between bodies (in the case of mere attention), or within a text input (in the case of self-attention)
Bias-Variance Trade-Off The notion that as a machine learning model includes more variance it begins to overfit the training data and as it includes more bias it begins to underfit the data, making the model less or more generalizable, or capable of predicting future values it was not trained on, in exchange for more accurately mapping to the training data
Classification Report A standard report for a classification machine learning model that includes precision, recall, and f1 score
Complete AI Solution Not just a model, ready to be trained, but also the complementary assets, like an extensive dataset to train the model for its purpose, and the business processes to collect that data, institutionalize it, fine tune the model, create oversight or human override procedures, and report on performance and key decisions
Complexity-Transparency Trade-Off The notion that as models get more complex, they become less transparent, and explainability must be exchanged for mere interpretability
Confusion Matrix An output of predicted data points as they relate to actual data points in the format of a matrix, resulting in an output of true positive count, false positive count, false negative count and true negative count
Deep Learning An AI algorithm with multiple layers of “neurons” between input and output; the opposite of transparent AI, as it, by definition, has multiple hidden layers
Dimensionality Reduction Algorithms or mathematical formulae that compress many features into groups of features for the purposes of a machine learning model; greatly reduces AI interpretability
F1 Score The harmonic mean of precision and recall to create a metric that seeks to balance the two types of error, but expresses them both
Foundational Trained with no task in mind, or with a dummy task, with the intention of having the model learn foundational knowledge, like language, then tuning that model to perform a specific task after the fact
Global Model Agnostic Methods AI explainability methods that give a comprehensive explanation of the entire dataset as it relates to the model, e.g., Permutation Importance, PDP, ALE
Gradient Boosting An algorithm that uses the output from one model as inputs for the next model, and each subsequent model in the chain focuses on correcting the previous model’s errors; increases complexity and reduces transparency
GridSearch An algorithm that will run a machine learning model multiple times with different predefined parameters to see which set of parameters result in the best model accuracy; does not reduce transparency
Layer Steps within a neural network that pass the outputs from one neuron as an input into the next neuron; multiple layers are often used in natural language processing neural networks, and some are often hidden from the developer, in the case of deep learning
Local Model Agnostic Methods AI explainability methods that aim to explain individual predictions of a model, e.g., ICE, SHAP, LIME
Loss Function The mathematical formula a machine learning model seeks to minimize in order to find the best option, e.g., a linear regression model seeks to minimize the least means squared loss function in order to find the line of best fit
Neural Network An AI algorithm that combines multiple neurons that perform mathematical functions on inputs to increase learnings, and pass this information as an output to the next neuron or to the final layer for decoding
Multi-headed Attention A variety of an attention mechanism that allows different “heads” within a natural language processing neural network to focus on different word associations at the same time
Overfitting The notion of training a model so specifically on one dataset that its learnings fail to be applicable generally, to the testing data set or to future data sets
Parameter A feature within a pre-built machine learning model that allows the data scientist to adjust how the model will be trained, e.g., number of clusters (k-means), number of estimators (random forests), C or gamma (support vector machines)
Precision A metric that shows how often a machine learning model predicts outcomes to be true, against how many are false positives; a model with a high precision is very trustworthy when it predicts a positive outcome
Pre-trained See “Foundational”
Purpose-built Not foundational, but built net-new for a specific task, and trained on only relevant data to help it accomplish that task
Recall A metric that shows how often a machine learning model predicts outcomes to be true against how many outcomes are actually true; a model with a high recall does not often miss an outcome that should be predicted positive
Self-Interpretable Self-interpretable models are, themselves, the explanations, and can be directly read and interpreted by a human, e.g., decision trees and regression models, including logistic regression
Supervised Learning Types of machine learning models that can compare predictions based on training data to actual testing data sets that were withheld from its training
Tokenization The process of reducing a word to a numerical code; an early step in most natural language processing algorithms
Transformer A broad category of neural networks that transform an input sequence into an output sequence, e.g., a sentence in english into a sentence in French, or a question into its answer, or a text-based prompt into a paper
Unsupervised Types of machine learning models or AI algorithms that learn without human intervention, e.g., clustering, some neural networks
Vectorization The process, used in most natural language processing algorithms, of compressing a word and all of its associated meanings or associations into a vector in space

Appendix B: Source Material

Recommended Content

Powered by Algolia AI Recommendations

Enable anyone to build great Search & Discovery

Get a demo
Start Free