# Hands-On Tutorial: Advanced Partitioning: Scenarios[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#hands-on-tutorial-advanced-partitioning-scenarios "Permalink to this headline")

Let’s create an automation scenario to rebuild the datasets in our Flow. One benefit of using a scenario to rebuild partitioned datasets in a Flow, is the availability of “keywords”. For example, instead of having to type the target partition identifier, we can use a keyword, such as “PREVIOUS\_DAY”.

By using keywords in a scenario, no matter what date is triggered by the scenario, Dataiku automatically computes the necessary partitions.

## Let’s Get Started![¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#let-s-get-started "Permalink to this headline")

In this tutorial, we will create a scenario that contains four steps, one for each dataset we want to build.

### Project Flow Overview[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#project-flow-overview "Permalink to this headline")

In this lesson, we will interact with both discrete and time-based partitioning inside scenarios. The goal of this lesson is to create a scenario that automatically specifies which partitions to build.

## Create the Project[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#create-the-project "Permalink to this headline")

From the Dataiku homepage, click **+New Project > DSS Tutorials > Advanced Designer > Advanced Partitioning: Scenarios (Tutorial)**.

Note

You can also download the starter project from this website and import it as a zip file.

### Explore the Flow[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#explore-the-flow "Permalink to this headline")

The Flow contains the following datasets:

* *cardholder\_info* contains information about the owner of the card used in the transaction process, such as FICO score and age.

* *merchant\_info* contains information about the merchant receiving the transaction amount, including the merchant subsector (*subsector\_description*).

* *transactions\_* contains historical information about each transaction, including the *purchase\_date*.

Note

A step in the Prepare recipe uses the “rand” function to simulate having new purchase date information in the transactions dataset. Since the expression, ‘-1 \* rand(0, 90)’, returns a random integer every time the computation runs, your results might differ from those shown in this tutorial. Visit the Formula language page in the product documentation for more information.

## Create a Scenario[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#create-a-scenario "Permalink to this headline")

In this section, we will focus on configuring our scenario to handle a Flow with partitioned datasets.

Note

To make it easier to view the Flow while building the scenario, you could open your project in two separate browser tabs.

### Build *transactions\_copy*[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#build-transactions-copy "Permalink to this headline")

* From the **Jobs** menu, select **Scenarios**.

* Click **Create Your First Scenario**.

* Name your scenario, `flow\_rebuild` and click **Create**.

Dataiku displays **Settings**. We could define a trigger here. However, in this lesson, we will use manual triggers.

* Click the **Steps** tab.

* Click **Add Step** and choose **Build / Train**.

* Click **Add Dataset to Build** and choose *transactions\_copy*.

Dataiku asks which partition we want to build. Since our dataset is partitioned using a time dimension, we can use the keyword, “PREVIOUS\_DAY”, as the target partition identifier. When the job runs, Dataiku uses the data prior to the current date as the target partition identifier. For more examples, visit Variables in Scenarios.

* In Partitioning: purchase\_date (YYYY-MM-DD), type the keyword, `PREVIOUS\_DAY`, then click Add.

To complete the configuration of this step, we need to specify the dataset build mode. We want to be able to test the scenario several times, and we want to be sure we include dependencies that go back to the start of the Flow. Even though it is computationally more expensive, we will select to force rebuild of the dataset and its dependencies for this initial step.

* In **Build mode**, select **Force-rebuild dataset and dependencies**.

* Save and Run the step.

While the job runs, Dataiku displays notifications informing us that the scenario started and that a job started. Let’s look at more details about this job.

We can see that Dataiku triggered a job in order to build the *transactions\_copy* dataset on the partition “PREVIOUS\_DAY”.

However, the second activity built “ALL” partitions. This is the expected since the Sync recipe that was used to create *transactions\_copy* was configured to use the Partition Redispatch feature. The Partition Redispatch feature writes all the available partitions in the Sync recipe’s output dataset.

### Build *transactions\_joined*[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#build-transactions-joined "Permalink to this headline")

* Return to your **flow\_rebuild** scenario and add another **Build / Train** step.

* Click **Add Dataset to Build** and choose *transactions\_joined*.

* Build the same partition as before: `PREVIOUS\_DAY`.

This time, we will keep the default build mode **Build required datasets**, since the smart reconstruction will be usable thanks to the execution of the first step we configured.

* **Save** and **Run** the step.

When we view the **Job**, we can see that the second step of the scenario targets the date of the “PREVIOUS\_DAY” (e.g., “2020-11-07”), as expected. This is more obvious when we view the **Last runs** of the “flow\_rebuild” scenario:

The benefit of using keywords in scenarios is no matter what date is triggered by the scenario, Dataiku automatically computes the necessary partitions.

The Flow is now built up until *transactions\_joined*.

### Build *transactions\_partitioned\_by\_sector*[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#build-transactions-partitioned-by-sector "Permalink to this headline")

Let’s now build *transactions\_partitioned\_by\_sector* which is partitioned by a discrete dimension, “merchant economic sector”.

* Return to your **flow\_rebuild** scenario and add a third **Build / Train** step.

* Click **Add Dataset to Build** and choose *transactions\_partitions\_by\_sector*.

* Build three partitions, `gas,internet,insurance`.

* Keep the default build mode **Build required datasets**.

* **Save** and **Run** the step.

* View the Job as it runs.

Once again, we can see that no matter which partitions we asked for in the scenario, the redispatching defined in the Sync recipe generates all the partitions in the output dataset. The behavior stays exactly the same as the one we observed with the time dimension.

Our Flow is now built through *transactions\_partitioned\_by\_sector*.

### Build *transactions\_known* and *transactions\_unknown*[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#build-transactions-known-and-transactions-unknown "Permalink to this headline")

Next, we will build the datasets output by the Split recipe, *transactions\_known* and *transactions\_unknown*.

* Return to your **flow\_rebuild** scenario and add a fourth **Build / Train** step.

* Click **Add Dataset to Build** and choose *transactions\_known*.

Since both “transactions\_known” and “transactions\_unknown” are computed by the Split recipe, we only need to choose one.

* Build three partitions, `gas,internet,insurance`.

* Keep the default build mode **Build required datasets**.

* **Save** and **Run** the step.

If we view the **Last run** for this step in the scenario, we can see that Dataiku has built the requested partitions, “gas”, “internet” and “insurance” for both *transactions\_known* and *transactions\_unknown*. Only the specified partitions were built–the Split recipe does not use the Partition Redispatch feature.

In this tutorial, the Flow is built until the prediction scoring recipe, leaving *transactions\_unknown\_scored* unbuilt.

## Summary[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#summary "Permalink to this headline")

Congratulations! You have completed this hands-on lesson. You now know how to manage a partitioned Flow in a scenario. Thanks to the keyword, “PREVIOUS\_DAY”, each day that you execute this scenario will result in new data, based on rows belonging to the partition of the previous day.

## What’s Next?[¶](https://knowledge.dataiku.com/latest/courses/advanced-partitioning/hands-on-scenario-partitioning.html#what-s-next "Permalink to this headline")

To challenge yourself further, you could try altering the steps in this tutorial to make this scenario work on all the partitions belonging to the 15 days before the current day without including the current day.
