# Model Evaluation Stores[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#model-evaluation-stores "Permalink to this headline")

Through the public API, the Python client allows you to perform evaluation of models. Those models are typically models trained in the Lab, and then deployed to the Flow as Saved Models (see Machine learning for additional information). They can also be external models.

## Concepts[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#concepts "Permalink to this headline")

### With a DSS model[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#with-a-dss-model "Permalink to this headline")

In DSS, you can *evaluate* a *version* of a *Saved Model* using an *Evaluation Recipe*. An Evaluation Recipe takes as input a Saved Model and a *Dataset* on which to perform this evaluation. An Evaluation Recipe can have three outputs:

* an *output* dataset,

* a *metrics* dataset, or

* a *Model Evaluation Store* (MES).

By default, the *active* version of the Saved Model is evaluated. This can be configured in the Evaluation Recipe.

If a MES is configured as an output, a *Model Evaluation* (ME) will be written in the MES each time the MES is built (or each time the Evaluation Recipe is run).

A Model Evaluation is a container for metrics of the evaluation of the Saved Model Version on the Evaluation Dataset. Those metrics include:

* all available *performance* metrics,

* the *Data Drift* metric.

The Data Drift metric is the accuracy of a model trained to recognize lines:

* from the evaluation dataset

* from the *train time test dataset* of the configured version of the Saved Model.

The higher this metric, the better the model can separate lines from the evaluation dataset from those from the train time test dataset. And so, the more data from the evaluation dataset is different from train time data.

Detailed information and other tools, including a binomial test, univariate data drift, and feature drift importance, are available in the *Input Data Drift* tab of a Model Evaluation. Note that this tool is interactive and that displayed results are not persisted.

### With an external model[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#with-an-external-model "Permalink to this headline")

In DSS, you can also evaluate an *external model* using a *Standalone Evaluation Recipe*. A *Standalone Evaluation Recipe* (SER) takes as input a labeled dataset containing labels, predictions, and (optionally) weights. A SER takes a single output: a Model Evaluation Store.

As the Evaluation Recipe, the Standalone Evaluation Recipe will output a Model Evaluation to the configured Model Evaluation Store each time it runs. In this case, however, the Data Drift can not be computed as there is no notion of reference data.

### How evaluation is performed[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#how-evaluation-is-performed "Permalink to this headline")

The Evaluation Recipe and its counterpart for external models, the Standalone Evaluation Recipe, perform the evaluation on a sample of the Evaluation Dataset. The sampling parameters are defined in the recipe. Note that the sample will contain at most 20,000 lines.

Performance metrics are then computed on this sample.

Data drift can be computed in three ways:

* at evaluation time, between the evaluation dataset and the train time test dataset;

* using the API, between the samples of a Model Evaluation, a Saved Model Version (sample of train time test dataset) or a Lab Model (sample of train time test dataset);

* interactively, in the “Input data drift” tab of a Model Evaluation.

In all cases, to compute the Data Drift, the sample of the Model Evaluation and a sample of the reference data are concatenated. In order to balance the data, those samples are truncated to the length of the smallest one. If the size of the reference sample if higher than the size of the ME sample, the reference sample will be truncated.

So:

* at evaluation time, we shall take as input the sample of the Model Evaluation (whose length is at most 20,000 lines) and a sample of the train time test dataset;

* interactively, the sample of the reference model evaluation and:

>

>

> 	+ if the other compared item is an ME, its sample;

> 	+ if the other compared item is a Lab Model or an SMV, a sample of its train time test dataset.

>

### Limitations[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#limitations "Permalink to this headline")

Model Evaluation Stores cannot be used with:

* clustering models,

* ensembling models,

* partitioned models.

Compatible prediction models have to be Python models.

## Usage samples[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#usage-samples "Permalink to this headline")

### Create a Model Evaluation Store[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#create-a-model-evaluation-store "Permalink to this headline")

§ # client is a DSS API client

§ p = client.get\_project("MYPROJECT")

§ mes\_id = p.create\_model\_evaluation\_store("My Mes Name")

Note that the display name of a Model Evaluation Store (in the above sample *My Mes Name*) is distinct from its unique id.

### Retrieve a Model Evaluation Store[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#retrieve-a-model-evaluation-store "Permalink to this headline")

§ # client is a DSS API client

§ p = client.get\_project("MYPROJECT")

§ mes\_id = p.get\_model\_evaluation\_store("mes\_id")

### List Model Evaluation Stores[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#list-model-evaluation-stores "Permalink to this headline")

§ # client is a DSS API client

§ p = client.get\_project("MYPROJECT")

§ stores =  p.list\_model\_evaluation\_stores(as\_type="objects")

### Create an Evaluation Recipe[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#create-an-evaluation-recipe "Permalink to this headline")

See `dataikuapi.dss.recipe.EvaluationRecipeCreator`

### Build a Model Evaluation Store and retrieve the performance and data drift metrics of the just computed ME[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#build-a-model-evaluation-store-and-retrieve-the-performance-and-data-drift-metrics-of-the-just-computed-me "Permalink to this headline")

§ # client is a DSS API client

§ p = client.get\_project("MYPROJECT")

§ mes = project.get\_model\_evaluation\_store("M3s\_1d")

§ mes.build()

§ me = mes.get\_latest\_model\_evaluation()

§ full\_info = me.get\_full\_info()

§ metrics = full\_info.metrics

### List Model Evaluations from a store[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#list-model-evaluations-from-a-store "Permalink to this headline")

§ # client is a DSS API client

§ p = client.get\_project("MYPROJECT")

§ mes = project.get\_model\_evaluation\_store("M3s\_1d")

§ me\_list = mes.list\_model\_evaluations()

### Retrieve an array of creation date / accuracy from a store[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#retrieve-an-array-of-creation-date-accuracy-from-a-store "Permalink to this headline")

§ p = client.get\_project("MYPROJECT")

§ mes = project.get\_model\_evaluation\_store("M3s\_1d")

§ me\_list = mes.list\_model\_evaluations()

§ res = []

§ for me in me\_list:

§ full\_info = me.get\_full\_info()

§ creation\_date = full\_info.creation\_date

§ accuracy = full\_info.metrics["accuracy"]

§ res.append([creation\_date,accuracy])

### Retrieve an array of label value / precision from a store[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#retrieve-an-array-of-label-value-precision-from-a-store "Permalink to this headline")

The date of creation of a model evaluation might not be the best way to key a metric. In some cases, it might be more interesting to use the labeling system, for instance to tag the version of the evaluation dataset.

If the user created a label “myCustomLabel:evaluationDataset”, he may retrieve an array of label value / precision from a store with the following snippet:

§ p = client.get\_project("MYPROJECT")

§ mes = project.get\_model\_evaluation\_store("M3s\_1d")

§ me\_list = mes.list\_model\_evaluations()

§ res = []

§ for me in me\_list:

§ full\_info = me.get\_full\_info()

§ label\_value = next(x for x in full\_info.user\_meta["labels"] if x["key"] == "myCustomLabel:evaluationDataset")

§ precision= full\_info.metrics["precision"]

§ res.append([label\_value,precision])

### Compute data drift of the evaluation dataset of a Model Evaluation with the train time test dataset of its base DSS model version[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#compute-data-drift-of-the-evaluation-dataset-of-a-model-evaluation-with-the-train-time-test-dataset-of-its-base-dss-model-version "Permalink to this headline")

§ # using base SMV is implicit

§ drift = me1.compute\_data\_drift()

§ drift\_model\_result = drift.drift\_model\_result

§ drift\_model\_accuracy = drift\_model\_result.drift\_model\_accuracy

§ print("Value: {} < {} < {}".format(drift\_model\_accuracy.lower\_confidence\_interval,

§ drift\_model\_accuracy.value,

§ drift\_model\_accuracy.upper\_confidence\_interval))

§ print("p-value: {}".format(drift\_model\_accuracy.pvalue))

### Compute data drift, display results and adjust parameters[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#compute-data-drift-display-results-and-adjust-parameters "Permalink to this headline")

§ # me1 and me2 are two compatible model evaluations (having the same prediction type) from any store

§ drift = me1.compute\_data\_drift(me2)

§ drift\_model\_result = drift.drift\_model\_result

§ drift\_model\_accuracy = drift\_model\_result.drift\_model\_accuracy

§ print("Value: {} < {} < {}".format(drift\_model\_accuracy.lower\_confidence\_interval,

§ drift\_model\_accuracy.value,

§ drift\_model\_accuracy.upper\_confidence\_interval))

§ print("p-value: {}".format(drift\_model\_accuracy.pvalue))

§ # Check sample sizes

§ print("Reference sample size: {}".format(drift\_model\_result.get\_raw()["referenceSampleSize"]))

§ print("Current sample size: {}".format(drift\_model\_result.get\_raw()["currentSampleSize"]))

§ # check columns handling

§ per\_col\_settings = drift.per\_column\_settings

§ for col\_settings in per\_col\_settings:

§ print("col {} - default handling {} - actual handling {}".format(col\_settings.name, col\_settings.default\_column\_handling, col\_settings.actual\_column\_handling))

§ # recompute, with Pclass set as CATEGORICAL

§ drift = me1.compute\_data\_drift(me2,

§ DataDriftParams.from\_params(

§ PerColumnDriftParamBuilder().with\_column\_drift\_param("Pclass", "CATEGORICAL", True).build()

§ )

§ )

§ ...

## API reference[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#api-reference "Permalink to this headline")

There are two main parts related to the handling of metrics and checks in Dataiku’s Python APIs:

* `dataiku.core.model\_evaluation\_store.ModelEvaluationStore` and `dataiku.core.model\_evaluation\_store.ModelEvaluation` in the dataiku package. They were initially designed for usage within DSS.

* `dataikuapi.dss.modelevaluationstore.DSSModelEvaluationStore` and `dataikuapi.dss.modelevaluationstore.DSSModelEvaluation` in the dataikuapi package. They were initially designed for usage outside of DSS.

Both set of classes have fairly similar capabilities.

For more details on the two packages, please see Python APIs.

### dataiku package API[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#dataiku-package-api "Permalink to this headline")

*class* `dataiku.``ModelEvaluationStore`(*lookup*, *project\_key=None*, *ignore\_flow=False*)

This is a handle to interact with a model evaluation store

Note: this class is also available as `dataiku.ModelEvaluationStore`

`get_info`(*sensitive\_info=False*)

Get information about the location and settings of this model evaluation store :rtype: dict

`get_path`()

Gets the filesystem path of this model evaluation store.

`get_id`()

`get_name`()

`list_runs`()

`get_evaluation`(*evaluation\_id*)

`get_last_metric_values`()

Get the set of last values of the metrics on this folder, as a `dataiku.ComputedMetrics` object

`get_metric_history`(*metric\_lookup*)

Get the set of all values a given metric took on this folder :param metric\_lookup: metric name or unique identifier

*class* `dataiku.core.model_evaluation_store.``ModelEvaluation`(*store*, *evaluation\_id*)

This is a handle to interact with a model evaluation

`set_preparation_steps`(*steps*, *requested\_output\_schema*, *context\_project\_key=None*)

`get_schema`()

Gets the schema of the sample in this model evaluation store, as an array of objects like this one: { ‘type’: ‘string’, ‘name’: ‘foo’, ‘maxLength’: 1000 }. There is more information for the map, array and object types.

`get_dataframe`(*columns=None*, *infer\_with\_pandas=True*, *parse\_dates=True*, *bool\_as\_str=False*, *float\_precision=None*)

Read the sample in the run as a Pandas dataframe.

Pandas dataframes are fully in-memory, so you need to make sure that your dataset will fit in RAM before using this.

Keywords arguments:

* infer\_with\_pandas – uses the types detected by pandas rather than the dataset schema as detected in DSS. (default True)

* parse\_dates – Date column in DSS’s dataset schema are parsed (default True)

* bool\_as\_str – Leave boolean values as strings (default False)

Inconsistent sampling parameter raise ValueError.

Note about encoding:

* Column labels are “unicode” objects

* When a column is of string type, the content is made of utf-8 encoded “str” objects

`iter_dataframes_forced_types`(*names*, *dtypes*, *parse\_date\_columns*, *sampling=None*, *chunksize=10000*, *float\_precision=None*)

`iter_dataframes`(*chunksize=10000*, *infer\_with\_pandas=True*, *parse\_dates=True*, *columns=None*, *bool\_as\_str=False*, *float\_precision=None*)

Read the model evaluation sample to Pandas dataframes by chunks of fixed size.

Returns a generator over pandas dataframes.

Useful is the sample doesn’t fit in RAM.

### dataikuapi package API[¶](https://doc.dataiku.com/dss/latest/api/python/model-evaluation-stores.html#module-dataikuapi.dss.modelevaluationstore "Permalink to this headline")

*class* `dataikuapi.dss.modelevaluationstore.``DSSModelEvaluationStore`(*client*, *project\_key*, *mes\_id*)

A handle to interact with a model evaluation store on the DSS instance.

Do not create this directly, use `dataikuapi.dss.DSSProject.get\_model\_evaluation\_store()`

*property* `id`

`get_settings`()

Returns the settings of this model evaluation store.

* Return type: DSSModelEvaluationStoreSettings

`get_zone`()

Gets the flow zone of this model evaluation store

* Return type: `dataikuapi.dss.flow.DSSFlowZone`

`move_to_zone`(*zone*)

Moves this object to a flow zone

* Parameters: **zone** (*object*) – a `dataikuapi.dss.flow.DSSFlowZone` where to move the object

`share_to_zone`(*zone*)

Share this object to a flow zone

* Parameters: **zone** (*object*) – a `dataikuapi.dss.flow.DSSFlowZone` where to share the object

`unshare_from_zone`(*zone*)

Unshare this object from a flow zone

* Parameters: **zone** (*object*) – a `dataikuapi.dss.flow.DSSFlowZone` from where to unshare the object

`get_usages`()

Get the recipes referencing this model evaluation store

* Returns:: a list of usages

`get_object_discussions`()

Get a handle to manage discussions on the model evaluation store

* Returns: the handle to manage discussions

* Return type: `dataikuapi.discussion.DSSObjectDiscussions`

`delete`()

Delete the model evaluation store

`list_model_evaluations`()

List the model evaluations in this model evaluation store. The list is sorted by ME creation date.

* Returns: The list of the model evaluations

* Return type: list of `dataikuapi.dss.modelevaluationstore.DSSModelEvaluation`

`get_model_evaluation`(*evaluation\_id*)

Get a handle to interact with a specific model evaluation

* Parameters: **evaluation\_id** (*string*) – the id of the desired model evaluation

* Returns: A `dataikuapi.dss.modelevaluationstore.DSSModelEvaluation` model evaluation handle

`get_latest_model_evaluation`()

Get a handle to interact with the latest model evaluation computed

* Returns: A `dataikuapi.dss.modelevaluationstore.DSSModelEvaluation` model evaluation handle
if the store is not empty, else None

`delete_model_evaluations`(*evaluations*)

Remove model evaluations from this store

`build`(*job\_type='NON\_RECURSIVE\_FORCED\_BUILD'*, *wait=True*, *no\_fail=False*)

Starts a new job to build this model evaluation store and wait for it to complete. Raises if the job failed.

§ job = mes.build()

§ print("Job %s done" % job.id)

* Parameters: * **job\_type** – The job type. One of RECURSIVE\_BUILD, NON\_RECURSIVE\_FORCED\_BUILD or RECURSIVE\_FORCED\_BUILD
* **wait** – wait for the build to finish before returning
* **no\_fail** – if True, does not raise if the job failed. Valid only when wait is True

* Returns: the `dataikuapi.dss.job.DSSJob` job handle corresponding to the built job

* Return type: `dataikuapi.dss.job.DSSJob`

`get_last_metric_values`()

Get the metrics of the latest model evaluation built

* Returns:: a list of metric objects and their value

`get_metric_history`(*metric*)

Get the history of the values of the metric on this model evaluation store

* Returns:: an object containing the values of the metric, cast to the appropriate type (double, boolean,…)

`compute_metrics`(*metric\_ids=None*, *probes=None*)

Compute metrics on this model evaluation store. If the metrics are not specified, the metrics setup on the model evaluation store are used.

*class* `dataikuapi.dss.modelevaluationstore.``DSSModelEvaluationStoreSettings`(*model\_evaluation\_store*, *settings*)

A handle on the settings of a model evaluation store

Do not create this class directly, instead use `dataikuapi.dss.DSSModelEvaluationStore.get\_settings()`

`get_raw`()

`save`()

*class* `dataikuapi.dss.modelevaluationstore.``DSSModelEvaluation`(*model\_evaluation\_store*, *evaluation\_id*)

A handle on a model evaluation

Do not create this class directly, instead use `dataikuapi.dss.DSSModelEvaluationStore.get\_model\_evaluation()`

`get_full_info`()

Retrieve the model evaluation with its performance data

* Returns: the model evaluation full info, as a `dataikuapi.dss.DSSModelEvaluationInfo`

`get_full_id`()

`delete`()

Remove this model evaluation

*property* `full_id`

`compute_data_drift`(*reference=None*, *data\_drift\_params=None*, *wait=True*)

Compute data drift against a reference model or model evaluation. The reference is determined automatically unless specified.

* Parameters: * **reference** (*Union**[**str**,* *DSSModelEvaluation**,* *DSSTrainedPredictionModelDetails**]*) – saved model version (full ID or DSSTrainedPredictionModelDetails)
or model evaluation (full ID or DSSModelEvaluation) to use as reference (optional)
* **data\_drift\_params** (*DataDriftParams*) – data drift computation settings as a `dataikuapi.dss.modelevaluationstore.DataDriftParams` (optional)
* **wait** – data drift computation settings (optional)

* Returns: a `dataikuapi.dss.modelevaluationstore.DataDriftResult` containing data drift analysis results if wait is True, or a `DSSFuture` handle otherwise

`get_metrics`()

Get the metrics for this model evaluation. Metrics must be understood here as Metrics in DSS Metrics & Checks

* Returns: the metrics, as a JSON object

`get_sample_df`()

Get the sample of the evaluation dataset on which the evaluation was performed

* Returns: the sample content, as a `pandas.DataFrame`

*class* `dataikuapi.dss.modelevaluationstore.``DSSModelEvaluationFullInfo`(*model\_evaluation*, *full\_info*)

A handle on the full information on a model evaluation.

Includes information such as the full id of the evaluated model, the evaluation params, the performance and drift metrics, if any, etc.

Do not create this class directly, instead use `dataikuapi.dss.DSSModelEvaluation.get\_full\_info()`

* 
`metrics`: The performance and data drift metric, if any.

* 
`creation_date`: The date and time of the creation of the model evaluation, as an epoch.

* 
`user_meta`: The user-accessible metadata (name, labels)
Returns the original object, not a copy. Changes to the returned object are persisted to DSS by calling `save\_user\_meta()`.

`get_raw`()

`save_user_meta`()

*class* `dataikuapi.dss.modelevaluationstore.``DataDriftParams`(*data*)

Object that represents parameters for data drift computation. Do not create this object directly, use `dataikuapi.dss.modelevaluationstore.DataDriftParams.from\_params()` instead.

*static* `from_params`(*per\_column\_settings*, *nb\_bins=10*, *compute\_histograms=True*, *confidence\_level=0.95*)

Creates parameters for data drift computation from columns, number of bins, compute histograms and confidence level

* Parameters: **per\_column\_settings** (*dict*) – A dict representing the per column settings.

You should use a `PerColumnDriftParamBuilder` to build it. :param int nb\_bins: (optional) Nb. bins in histograms (apply to all columns) - default: 10 :param bool compute\_histograms: (optional) Enable/disable histograms - default: True :param float confidence\_level: (optional) Used to compute confidence interval on drift’s model accuracy - default: 0.95

* Return type: `dataikuapi.dss.modelevaluationstore.DataDriftParams`

*class* `dataikuapi.dss.modelevaluationstore.``PerColumnDriftParamBuilder`

Builder for a map of per column drift params settings. Used as a helper before computing data drift to build columns param expected in `dataikuapi.dss.modelevaluationstore.DataDriftParams.from\_params()`.

`build`()

Returns the built dict for per column drift params settings

`with_column_drift_param`(*name*, *handling='AUTO'*, *enabled=True*)

Sets the drift params settings for given column name.

* Param: string name: The name of the column

* Param: string handling: (optional) The column type, should be either NUMERICAL, CATEGORICAL or AUTO (default: AUTO)

* Param: bool enabled: (optional) False means the column is ignored in drift computation (default: True)

*class* `dataikuapi.dss.modelevaluationstore.``DataDriftResult`(*data*)

A handle on the data drift result of a model evaluation.

Do not create this class directly, instead use `dataikuapi.dss.DSSModelEvaluation.compute\_data\_drift()`

* 
`drift_model_result`: Drift analysis based on drift modeling.

* 
`univariate_drift_result`: Per-column drift analysis based on pairwise comparison of distributions.

* 
`per_column_settings`: Information about column handling that has been used (errors, types, etc).

`get_raw`()

* Returns: the raw data drift result

* Return type: dict

*class* `dataikuapi.dss.modelevaluationstore.``DriftModelResult`(*data*)

A handle on the drift model result.

Do not create this class directly, instead use `dataikuapi.dss.modelevaluationstore.DataDriftResult.drift\_model\_result`

`get_raw`()

* Returns: the raw drift model result

* Return type: dict

*class* `dataikuapi.dss.modelevaluationstore.``UnivariateDriftResult`(*data*)

A handle on the univariate data drift.

Do not create this class directly, instead use `dataikuapi.dss.modelevaluationstore.DataDriftResult.univariate\_drift\_result`

* 
`per_column_drift_data`: Drift data per column, as a dict of column name -> drift data.

`get_raw`()

* Returns: the raw univariate data drift

* Return type: dict

*class* `dataikuapi.dss.modelevaluationstore.``ColumnSettings`(*data*)

A handle on column handling information.

Do not create this class directly, instead use `dataikuapi.dss.modelevaluationstore.DataDriftResult.get\_per\_column\_settings()`

* 
`actual_column_handling`: The actual column handling (either forced via drift params or inferred from model evaluation preprocessings).
It can be any of NUMERICAL, CATEGORICAL, or IGNORED.

* 
`default_column_handling`: The default column handling (based on model evaluation preprocessing only).
It can be any of NUMERICAL, CATEGORICAL, or IGNORED.

`get_raw`()

* Returns: the raw column handling information

* Return type: dict

*class* `dataikuapi.dss.modelevaluationstore.``DriftModelAccuracy`(*data*)

A handle on the drift model accuracy.

Do not create this class directly, instead use `dataikuapi.dss.modelevaluationstore.DriftModelResult.drift\_model\_accuracy`

`get_raw`()

* Returns: the drift model accuracy data

* Return type: dict
