# AML Alerts Triage[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#aml-alerts-triage "Permalink to this headline")

Contents

* Overview

+ Business Case

+ Technical Requirements

+ Installation

+ Data Requirements

+ Workflow Overview

* Walkthrough

+ Gather Input Data and Prepare for Training

+ Training an Alert Triage Model

+ Using a Rules-Based Approach for Alerts Triage

+ Long Term Model Performance

+ Enable Compliance Teams with Pre-Built Explainability Dashboards

+ Reproducing these Processes With Minimal Effort For Your Own Data

## Overview[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#overview "Permalink to this headline")

### Business Case[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#business-case "Permalink to this headline")

Anti-money laundering processes are complex and multifaceted, and generate large numbers of alerts which must be investigated. Most generated alerts are ultimately not escalated for further review. Reducing the total number of false-positive alerts is a complex and heavily regulated process. Improvements in AML processes must occur at many points in the chain, and a modular solution that can be readily incorporated into existing flows to more efficiently process existing alerts is a means to improve detection rates and reduce alert fatigue, acting as a first step to AML set-up efficiency. Thanks to this adapt and apply Solution, Financial Crime analysts are supported in initial assessment through risk likelihood prioritization. Insights delivered by the Solution also include other elements which can be used as a starting point to review effectiveness of used business rules, paving the road to further AML set-up reinforcement.

### Technical Requirements[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#technical-requirements "Permalink to this headline")

To leverage this solution, you must meet the following requirements:

* Have access to a Dataiku 10.0.5+ instance.

The downloadable version uses filesystem managed datasets and the built-in Dataiku engine as the only processing engine. Performance could be greatly improved by changing all the connections to Snowflake connections. In particular, daily batch computation time can be optimized from 2 hours to a couple of minutes by switching from File System managed datasets to Snowflake connections and enabling the use of an in-database SQL engine.

Note

Due to the deprecation of the the Model Drift Plugin, we’ve upgrade the solution to use the Model Evaluation Store native feature which is available only on Dataiku 10.0+ instances. As a result, the previous version of this solution is available on Dataiku 9.0 but will no longer be supported or updated. We recommend users to upgrade to the latest version of Dataiku to take full advantage of this solution as well as new and exciting features.

### Installation[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#installation "Permalink to this headline")

If the technical requirements are met, this solution can be installed in one of two ways:

* On your Dataiku instance click **+ New Project** > **Business solutions** > Search for **AML Alert Triage**.

* Download the .zip project file and upload it directly to your Dataiku instance as a new project.

### Data Requirements[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#data-requirements "Permalink to this headline")

We will work with a fictional financial services company called Haiku Bank, and use their data to illustrate the steps to prioritize AML alerts. In this project, alerts are defined at a transaction level so within our initial alerts datasets we will find transaction ids, alert ids, and the label is\_escalated for historical alerts. Additionally, we have a *segments* input dataset which represents client categories that have been defined using KYC.

Note

This project is meant to be used as a template to guide development of your own analysis on Dataiku’s platform. The results of the model should not be used as actionable insights and the data provided with the project may not be representative of actual data in a real-life project.

### Workflow Overview[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#workflow-overview "Permalink to this headline")

You can follow along with the solution in the Dataiku gallery.

The project has the following high level steps:

Join and prepare our input data

Train an alert triage model

Conduct rules based triage in parallel

Monitor the performance of our model over time

Enable compliance teams to understand the models they use via interactive Model Explainability in a pre-built Dashboard

Automate the full pipeline to react to score new alerts, monitor model performance over time, and retrain the model with new data

## Walkthrough[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#walkthrough "Permalink to this headline")

Note

In-depth technical details can be found in the wiki of the project.

### Gather Input Data and Prepare for Training[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#gather-input-data-and-prepare-for-training "Permalink to this headline")

The first two flow zones of our project are fairly straightforward. We begin in the **Input** flow zone by bringing in 6 initial datasets:

* *alerts\_history* is combined with *new\_alerts* via a stack recipe. All of the alerts in the *new\_alerts* are obviously missing values for the **is\_escalated** column since they haven’t yet been classified.

* Similarly, *transactions\_history* and *new\_transactions* are stacked together. New transactions correspond to the last day’s transactions but both datasets share the same schema.

* This zone also combines client information (*accounts*) with *segments* representing KYC created client categories.

This project has scenarios that can be configured to be triggered based on changes to these initial datasets but some additional work is needed to update the project to connect to updating datasets. Within the first flow zone, some additional data preparation is done to handle email addresses, ip addresses, and home addresses.

Moving along to the **Data Preparation** flow zone, we sequentially process the data to create some additional features for the model’s feature engineering. A first prepare recipe creates the age of the customer and then a window recipe creates all the major features that will be important for the model. The data is additionally filtered to keep only alerts before being split into three datasets:

* *transactions\_train*: contains the bulk of the data

* *transactions\_test*: contains the last 4 weeks of data (the split is done automatically via a project variable set by the scenario **Evaluate Model**)

* *transactions\_unlabelled*: contains the unlabelled transactions from the last day

### Training an Alert Triage Model[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#training-an-alert-triage-model "Permalink to this headline")

The **Alert Triage** flow zone takes as an input two datasets: *alerts\_train* and *alerts\_unlabelled*. We train a two-class classification model that predicts the previously mentioned **is\_escalated** variable. The dataset is imbalanced with around 4% of 1 and the rest of 0 and handled using class weights as a weighting strategy. All variables that make sense are included in the model, and processed in a standard way (dummy encoded for categories and normalized for numerical variables). In the design part of the model, we choose a custom cost matrix function to optimize the threshold. Thus, false negatives are heavily weighted, and true positive and false positive have the same weight but with opposite signs. Therefore, the user can input how important each of the metrics is within the parametrization of the model.

We ultimately selected the XGBoost algorithm because it shares the same performance as the Random Forest algorithm while being lighter. Looking at the variable importance of our model, **orig\_is\_escalated\_avg** is the most important variable, which makes sense because past escalated alerts influence new alerts being escalated.

The deployed model is then used to score the new alerts and the main output from this scoring is the **proba\_1** column which can be interpreted as a priority. Compliance officers would first investigate the high priority alerts before processing the ones further down the list. Thus the scored alerts are sorted by priority and represent the output of the project that will be used elsewhere.

### Using a Rules-Based Approach for Alerts Triage[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#using-a-rules-based-approach-for-alerts-triage "Permalink to this headline")

In parallel to the Machine Learning model for Alert triaging, we also generate some rules-based priority scores for new alerts in the **Rules Based Triage** flow zone. There are two parallel branches in this flow zone. In the first branch a prepare recipe uses a predefined formula in a prepare recipe on the *transactions\_unlabelled* dataset. The full dataset is then sorted based on the resulting priority score to place alerts with the highest priority score first.

In the second branch, we once again use the same predefined formula in a prepare recipe on the *alerts\_test* dataset in order to evaluate the performance of this rule and compare it to the machine learning model. As we are using a prepare recipe and not a visual ML model to prioritize alerts, we cannot use the evaluate recipe to compute metrics. Instead we apply the same rule as for the unlabelled alerts for the test alerts and compute the needed aggregates using the grouping recipe. And finally the false positive and false negative rates are computed before appending results each time a new evaluation is executed.

### Long Term Model Performance[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#long-term-model-performance "Permalink to this headline")

We end with the **Model Drift** flow zone where the model’s performance over time is monitored, as well as the data, to ensure neither has drifted too much. Using the *alerts\_test* dataset, we compute the performance metrics of the model on data made of the last 4 weeks of alerts and has not been used in the model training. As time goes by, the model will become more and more outdated with regards to the updated test data. We use a model evaluation recipe to compute all the standard metrics to evaluate the performance of a binary classification model. A prepare recipe removes unnecessary metrics and creates the ones that are most relevant to the business, namely the false positive rate and the false negative rate. The two subsequent recipes pick the oldest and the most recent evaluations and then compute the difference between them for each of these values. We also compare the performance metrics from the rules-based prioritization with the ML-based prioritization. A trigger can be created on the output of the model drift performance to retrain the model when it has drifted too much.

In the upper part of the flow zone, we also conduct data drift analysis. First, the *alerts\_validation* dataset is created in the same way as in the model. We then leverage the evaluate recipe from the Model Evaluation Store to compute the data drift between the validation dataset used for the training at the moment when the model was deployed and the most recent validation dataset. So at the first iteration, there is no data drift because the two datasets are identical. Then as data drifts over time, the drift model accuracy increases, meaning that it is getting easier to discriminate between the two datasets. When the drift exceeds 0.5, the user should start worrying about the drift and consider training again with fresher data.

### Enable Compliance Teams with Pre-Built Explainability Dashboards[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#enable-compliance-teams-with-pre-built-explainability-dashboards "Permalink to this headline")

A key component of the AML process is the ability of Compliance Teams to understand precisely how the models they are using behave and ensure the models won’t break under certain conditions. In the pre-built project dashboard there are several components that can be used to investigate the trained model.

In the Global Explanation tab of the dashboard, we can analyze our model via feature importance analysis and partial dependence plots. If there are surprising results in the importance or dependence values for certain variables then investigation into the model is warranted. By enabling Compliance Teams to have access to these graphs themselves, it empowers them to use their invaluable knowledge to additionally evaluate model performance.

The Individual Explanations tab allows us to dig a bit deeper on the model performance. Users of the Dashboard can interactively change model features and see the output from the model. Additionally, users can see how probability and influential features are impacted by their changes to the model feature values. A final view in this dashboard focused on individual explanations can be used to detect patterns in how the top and bottom predictions have been made.

As previously mentioned, model and data drift should be continuously monitored. This can be done using auto-trigger scenarios for re-training, the Drift tab of the dashboard (where the model is evaluated each time new data comes in through the Model Evaluation Store), and the results view of our saved model.

### Reproducing these Processes With Minimal Effort For Your Own Data[¶](https://knowledge.dataiku.com/latest/kb/business-solutions/aml-alert-triage/aml-alert-triage.html#reproducing-these-processes-with-minimal-effort-for-your-own-data "Permalink to this headline")

The intent of this project is to enable compliance teams to understand how Dataiku can be used to supplement their existing AML processes by prioritizing alerts. By creating a singular solution that can benefit and influence the decisions of a variety of teams in a single organization, smarter and more holistic strategies can be designed in order to prioritize investigations, avoid additional regulatory burden, and provide insights for reviewing business rules.

We’ve provided several suggestions on how to use alert data to classify and triage alerts but ultimately the “best” approach will depend on your specific needs and your data. If you’re interested in adapting this project to the specific goals and needs of your organization, roll-out and customization services can be offered on demand.
