# Concept: Where does the computation happen?[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#concept-where-does-the-computation-happen "Permalink to this headline")

In the Dataiku flow, you take datasets from different sources (SQL, file-system, HDFS) and you seamlessly apply recipes (like SQL queries, preparation scripts or computing predictions from a model). But what happens under the hood, where does the computation actually take place? In this post, we will present you succinctly the architecture behind DSS, and explain why it matters.

## Overview[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#overview "Permalink to this headline")

Computation in Dataiku can take four main forms:

* in-memory

* streamed

* runs in database (SQL)

* runs on-Hadoop.

Which strategy is used depends on the dataset and the operation you are applying to the dataset.

Why does it matter? Well, sometimes if you apply the wrong type of recipe to your dataset, your computation will be painfully long or run out of memory! These issues particularly arise with big data.

Here is a table summarizing where the computation takes place for each operation performed in Dataiku:

|  | Locally in DSS | In Hadoop / Spark | In SQL Database | In Kubernetes / Docker |

| --- | --- | --- | --- | --- |

| Visual Preparation Design | In-memory Sample | N/A | N/A | N/A |

| Visual Preparation Execution | YES Streaming | YES Spark | YES | N/A |

| Visual Recipes (other than Prepare) | YESStreaming or disk-copy | YESHive, Spark, Impala | YES | N/A |

| Python and R recipes | YESIn-memory or streaming | YESPySpark, SparkR, sparklyr | Custom code with DSS helper API | YESIn-memory or streaming |

| Spark-Scala recipe | N/A | YES | N/A | N/A |

| Charts | YES | YESHive, Impala (most charts) | YES(most charts) | N/A |

| Machine Learning train | YESscikit-learn, XGBoost, Keras/Tensorflow | YESMLlib, Sparkling Water | YES  | YESscikit-learn, XGBoost, Keras/Tensorflow |

| Machine Learning execution | YESscikit-learn, XGBoost, MLlib, Keras/Tensorflow | YESscikit-learn, XGBoost, MLlib, Sparkling Water | YESscikit-learn (partial), XGBoost, MLlib (some models) | YESscikit-learn, XGBoost, MLlib, Keras/Tensorflow |

| Python, R, Scala notebooks | YESIn-memory or streaming | YESSpark-Scala, PySpark, SparkR, sparklyr | Custom code with DSS helper API | YES |

| SQL-like recipe or notebook | N/A | YESHive, Impala, SparkSQL | YES | N/A |

Some rules are clear and easy to understand. Take a SQL recipe. First, remember that the input datasets of your SQL recipe must be in a SQL database. When you hit run in the SQL recipe, Dataiku will send a query to the SQL database: read the input datasets, perform the SQL query, and finally write the output dataset if it is a SQL dataset, or streams the output otherwise.

So the computation actually runs in-database. The idea is the same for Hive recipes, but here is applied to HDFS datasets, and everything runs in-cluster.

Dataiku does not solely act as a server, sending queries to a SQL-database or a Hadoop-cluster. It also performs computation, what we called the Dataiku engine, which uses two main strategies:

* in-memory; the data is stored in RAM

* streamed; Dataiku reads the input dataset as a stream of rows, applies computations to the rows as they arrive, and writes the output datasets row per row.

Note

The Dataiku engine also uses a third strategy for charts, but this a particular topic that is well-explained in the documentation.

## In-memory processing[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#in-memory-processing "Permalink to this headline")

By concept, in-memory computation can lead to out-of-memory situations on large datasets.

When can this annoying situation actually happen in Dataiku? As you can see in the above table, in-memory computation can happen in the following situations:

* Python and R recipes

* Editing a preparation recipe (or analysis)

* Training a machine learning model

### Visual data preparation[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#visual-data-preparation "Permalink to this headline")

When you are editing a preparation script (a Prepare recipe), Dataiku actually **samples the dataset** to ensure that the design computations will fit in RAM.

In a preparation script, you can add any processor and the editor is always fast and responsive, no matter the size of the original dataset because the computation is done only on a small sample.

Note that you can change the sampling, and you could even use the whole dataset. But you should really think twice before switching the sampling to the whole dataset when you have a large dataset.

First of all, it misses the point of having a fast and responsive editing mode, and you may even run out of memory!

Warning

**In a preparation script, do not use a design sample that is too large.**

### Data preparation execution[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#data-preparation-execution "Permalink to this headline")

You may have noticed that the strategy used in preparation script execution (when you click run) is completely different from the one used in editing the script.

Dataiku will choose the best strategy available (streamed or in-cluster) to execute the script, but never in-memory. Dataiku will of course apply the computation to the whole dataset, which is why it does not use in-memory processing.

### Python and R recipes[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#python-and-r-recipes "Permalink to this headline")

When you are using the pandas Python package, or R, you should keep in mind that both require **in-memory data manipulation**. As such, **R and pandas are not well-suited tools for large data manipulation**. If the input or output dataset cannot fit in RAM, you may run out of memory.

Warning

**You cannot apply R recipes or Python recipes with pandas to datasets that are too big.**

In Python, several workarounds are available:

* Use streamed (per-row) processing instead of Dataframe processing

* Use chunked processing (read dataframes by blocks)

Note

The product documentation has details on these implementations in Python recipes.

## How to deal with big data?[¶](https://knowledge.dataiku.com/latest/kb/data-prep/where-compute-happens.html#how-to-deal-with-big-data "Permalink to this headline")

Python+Pandas or R remain useful when working with big data.

For example, sometimes, you can limit yourself to a sample of the dataset. It’s a good way to prototype your code. Dataiku allows you to stream a random sample of the data and import it directly as a Pandas dataframe (by passing the sampling and limit arguments in the `get\_dataframe()` function) or R dataframe (by passing the sampling argument in the `read.dataset()` function).

But afterwards, if you want to apply the computation to the whole dataset, you will have to switch to another computing strategy.

Dataiku offers many alternatives:

* As mentioned, you could use a streamed implementation of your Python code

* A **Python UDF** processor in a preparation recipe could work

* In some cases, using a SQL recipe may be a very good option.

However, when dealing with big data, you want to **push the computation to the data** as much as possible. Computing engines like Hadoop on HDFS follow this principle, and are specifically designed for big-data computing. Hive recipes, which run in-cluster, are well-suited for big data computing.

Dataiku allows you to easily switch between computing engines. For instance, suppose your current dataset is on the filesystem, but is too large for a python recipe. You can use a Sync recipe to push your dataset to HDFS, and opt for a Hive or PySpark recipe instead.
