# Concept: Computation Engine[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#concept-computation-engine "Permalink to this headline")

Tip

This content is also included in the free Dataiku Academy course, Basics 102, which is part of the Core Designer learning path. Register for the course there if you’d like to track and validate your progress alongside concept videos, summaries, hands-on tutorials, and quizzes.

Computation has a cost. In Dataiku DSS, there are computation strategies that help with reducing this cost. Dataiku can perform the computation using the DSS engine or push down the computation to external engines. Dataiku acts as an orchestrator of your connections’ engines, delegating the computation to these connections when possible.

When transforming your dataset, you are actually working with a sample of it. When you are ready to apply transformation steps to the whole dataset, click **Run**. Then, you can select a computation engine that best fits the underlying data storage and the operation you are applying to the dataset.

## Recipe Engines[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#recipe-engines "Permalink to this headline")

Computation in Dataiku can take four main forms. To avoid running out of memory when manipulating large datasets, Dataiku recommends offloading the computation to where the data lives. This way you avoid bringing all the data into memory or streaming it through the DSS server.

## In DSS Engine: In-memory or Streamed[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#in-dss-engine-in-memory-or-streamed "Permalink to this headline")

For in-memory execution, the data are stored in RAM. This strategy is used, for example, to execute Python or R recipes.

For streamed execution, Dataiku reads the input dataset as a stream of rows, applies computation to the rows as they arrive, and writes the output datasets row per row.

## In-database[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#in-database "Permalink to this headline")

This strategy is used, for example, to execute SQL queries. The visual recipe is translated into an SQL query, which is then injected, or pushed down, to the SQL server.

## On Hadoop/Spark Cluster[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#on-hadoop-spark-cluster "Permalink to this headline")

Depending on the engine you choose, the visual recipe is translated into a Hive, Impala, or Spark SQL query, which is then injected, or pushed down, to the Hadoop or Spark cluster.

## In Kubernetes/Docker[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#in-kubernetes-docker "Permalink to this headline")

This strategy is also in-memory or streamed but uses container execution through Docker and Kubernetes clusters rather than the Dataiku host server.

## Learn More[¶](https://knowledge.dataiku.com/latest/courses/basics/explore-flow/concept-computation-engine.html#learn-more "Permalink to this headline")

In this lesson, you learned about ways to optimize computation when working in Dataiku DSS. Continue learning about the Basics of Dataiku DSS by visiting Concept: Jobs.
