# MLLib (Spark) engine[¶](https://doc.dataiku.com/dss/latest/machine_learning/mllib.html#mllib-spark-engine "Permalink to this headline")

MLLib is Spark’s machine learning library. DSS can use it to train prediction or clustering models on your large datasets that don’t fit into memory.

Warning

Spark’s overhead is non-negligible and its support is limited (see Limitations).     If your data fits into memory, you should use regular in-memory ML instead for faster learning and more extensive options and algorithms.

* Usage

* Prediction Algorithms

* Clustering algorithms

* Custom Models

* Limitations

## Usage[¶](https://doc.dataiku.com/dss/latest/machine_learning/mllib.html#usage "Permalink to this headline")

When you create a new machine learning in an Analysis, you can select the backend. By default it’s Python in memory, but if you have Spark correctly set up, you can also see Spark MLLib. Select it and your model will be trained on Spark, using algorithms available in MLLib or your custom MLLib-compatible models.

You can then fine-tune your model, deploy it in the Flow as a retrainable model and apply it in a scoring recipe to perform prediction on unlabelled datasets. Clustering models may also be retrained on new datasets through the cluster recipe.

In the model’s settings and the training, scoring and cluster recipes, there is an additional Spark config section, in which you can:

* Change the base Spark configuration

* Add / override Spark configuration options

* Select the storage level of the dataset for caching once the data is loaded

and prepared

* Select the number of Spark RDD partitions to split non-HDFS input datasets

See DSS and Spark for more information about Spark in Data Science Studio.

## Prediction Algorithms[¶](https://doc.dataiku.com/dss/latest/machine_learning/mllib.html#prediction-algorithms "Permalink to this headline")

DSS 11 supports the following algorithms on MLLib:

* Logistic Regression (classification)

* Linear Regression (regression)

* Decision Trees (classification & regression)

* Random Forest (classification & regression)

* Gradient Boosted Trees (binary classification & regression)

* Naive Bayes (multiclass classification)

* Custom models

## Clustering algorithms[¶](https://doc.dataiku.com/dss/latest/machine_learning/mllib.html#clustering-algorithms "Permalink to this headline")

DSS 11 supports the following algorithms on MLLib:

* KMeans (clustering)

* Gaussian Mixtures (clustering)

* Custom models

## Custom Models[¶](https://doc.dataiku.com/dss/latest/machine_learning/mllib.html#custom-models "Permalink to this headline")

Models using custom code may be trained with the MLLib backend. To train such a model,

* Implement classes extending the `Estimator` and `Model` classes of the `org.apache.spark.ml`

package. These will be the classes used to train your model: DSS will call the `fit(DataFrame)`

method of your `Estimator` and the `transform(DataFrame)` method of your `Model`.

* Package your classes and all necessary classes in a jar, and place it in the `lib/java` folder

of your data directory.

* In DSS, open your MLLib model settings and add a new custom algorithm in the algorithm list.

* Place the initialization (scala) code for your `Estimator` into the code editor, together with

any necessary `import` statements. The initialization statement should be the last to be called.

Note that declaring classes (including anonymous classes) in the editor is not recommended, as it

may cause serialization errors. They should therefore be compiled and put in the jar.

## Limitations[¶](https://doc.dataiku.com/dss/latest/machine_learning/mllib.html#limitations "Permalink to this headline")

On top of the general Spark limitations in DSS, MLLib has specific limitations:

* Gradient Boosted Trees in MLLib does not output per-class

probabilities, so there is no threshold to set, and some metrics (AUC, Log

loss, Lift) are not available, as are some report sections (variable

importance, decision & lift charts, ROC curve).

* Some feature preprocessing options are not available (although most can be achieved

by other means):

+ Feature combinations

+ Numerical handling other than regular

+ Categorical handling other than dummy encoding

+ Text handling other than Tokenize, hash & count

+ Dimensionality-reduction for clustering

* If test dataset is larger than 1 million rows, it will be subsampled to ~1M

rows for performance and memory consumption reasons, since some scoring

operations require sorting and collecting the whole data.

* K-fold cross-test and hyperparameter optimization (grid search) are not supported.

* Build partitioned models with MLLib backend is not supported

* Post training computations like partial dependence plot and subpopulation analysis are not supported

* Individual explanations are not supported

* Containerized execution is not supported

* Debugging capabilities such as Assertions and Dignostics are not supported

* Interactive scoring is not available
