# Hands-On Tutorial: The Basics of Python in Dataiku[¶](https://knowledge.dataiku.com/latest/courses/advanced-code/python/basics.html#hands-on-tutorial-the-basics-of-python-in-dataiku "Permalink to this headline")

Dataiku allows you to seamlessly integrate Python code and visual recipes in a Flow.

In this tutorial, we will show you how to:

* Integrate Python as part of your data pipeline through code recipes.

* Use Jupyter notebooks to prototype and test code.

* Transfer a Dataiku dataset into a pandas dataframe and back, using the `dataiku` Python package.

We will work with the fictional retailer Haiku T-Shirt’s data.

## Prerequisites[¶](https://knowledge.dataiku.com/latest/courses/advanced-code/python/basics.html#prerequisites "Permalink to this headline")

This tutorial assumes that you are familiar with the Dataiku Basics courses.

## Create Your Project[¶](https://knowledge.dataiku.com/latest/courses/advanced-code/python/basics.html#create-your-project "Permalink to this headline")

The first step is to create a new Dataiku Project.

* From the homepage, click **+New Project > DSS Tutorials > Developer > Code Notebooks (Tutorial)**.

Note

You can also download the starter project from this website and import it as a zip file.

In the Flow, you see the Haiku T-Shirt *orders* and *customers\_stacked* data uploaded into Dataiku. Further, the *customers\_stacked* data has been prepared with a visual **Prepare** recipe.

## Your First Python Recipe[¶](https://knowledge.dataiku.com/latest/courses/advanced-code/python/basics.html#your-first-python-recipe "Permalink to this headline")

Our current goal is to group past orders by customer, aggregating their past interactions. In the Basics courses, we accomplished this with a visual **Group** recipe, but it can also be easily accomplished with Python code.

* With the *orders* dataset selected, choose **Actions > Code Recipes > Python**.

* Add a new output dataset named `orders\_by\_customer`.

* Click **Create Recipe**.

The recipe editor is now populated with the following code, which:

* reads the *orders* dataset into a pandas dataframe named *orders\_df*,

* passes it unchanged to a new dataframe named *orders\_by\_customer\_df*, and

* writes that new dataframe out to the *orders\_by\_customer* dataset.

§ # -\*- coding: utf-8 -\*-

§ import dataiku

§ import pandas as pd, numpy as np

§ from dataiku import pandasutils as pdu

§ # Read recipe inputs

§ orders = dataiku.Dataset("orders")

§ orders\_df = orders.get\_dataframe()

§ # Compute recipe outputs from inputs

§ # TODO: Replace this part by your actual code that computes the output, as a Pandas dataframe

§ # NB: DSS also supports other kinds of APIs for reading and writing data. Please see doc.

§ orders\_by\_customer\_df = orders\_df # For this sample code, simply copy input to output

§ # Write recipe outputs

§ orders\_by\_customer = dataiku.Dataset("orders\_by\_customer")

§ orders\_by\_customer.write\_with\_schema(orders\_by\_customer\_df)

As the commented TODO says, we’ll need to provide the code that aggregates the orders by customer. Dataiku provides a number of **code samples** to help get us started.

* In this case, replace the line that defines `orders\_by\_customer\_df` with the following code.

§ orders\_by\_customer\_df = orders\_df.assign(total=orders\_df.tshirt\_price\*orders\_df.tshirt\_quantity

§ ).groupby(by="customer\_id"

§ ).agg({"pages\_visited":"mean",

§ "total":"sum"})

This creates a dataframe with rows grouped by `customer\_id`. For each customer, we’ve computed the average number of pages on the Haiku T-shirt website visited by the customer during orders, and the sum total of the value of orders made by the customer, where the value of each order is the price of each t-shirt times the number of t-shirts purchased.

* Now run the recipe, and when it completes, explore the output dataset.

Interestingly, the output dataset does not contain the *customer\_id* column, but we’ll need it later in order to join the customers and grouped orders data.

* To diagnose the issue, in the *orders\_by\_customer* dataset click **Parent Recipe** to reopen the Python recipe.

* Then click **Edit in Notebook**.

This opens a Jupyter notebook with the recipe code, where we can interactively test it. The recipe code begins in a single cell.

* Split the cell so that the code to write recipe outputs is in a separate cell.

* Add a cell between the two existing cells and put the following code in it.

§ orders\_by\_customer\_df.head()

* Run the first two cells.

The output shows that the *orders\_by\_customer* dataframe has the *customer\_id* information; however, the dataframe has a hierarchical index.

* In order to flatten the index, add `.reset\_index()` to the code that defines the dataframe so that it looks like the following.

§ orders\_by\_customer\_df = orders\_df.assign(total=orders\_df.tshirt\_price\*orders\_df.tshirt\_quantity

§ ).groupby(by="customer\_id"

§ ).agg({"pages\_visited":"mean",

§ "total":"sum"}).reset\_index()

* Re-run the first two cells to see how the dataframe has changed.

* Click **Save back to recipe** and run the recipe again.

Now the output dataset contains a *customer\_id* column.

## Explore with a Python Notebook[¶](https://knowledge.dataiku.com/latest/courses/advanced-code/python/basics.html#explore-with-a-python-notebook "Permalink to this headline")

Previously, we started with a Python recipe because we had a specific goal of transforming the *orders* dataset. Alternatively, we can explore a dataset using a notebook.

* In the Flow, select the *customers\_stacked\_prepared* dataset.

* From the Actions tab, click **Lab > New Code Notebook > Python**.

* Read the dataset in memory using Pandas.

* Click **Create**.

The notebook is automatically populated with four cells.

The first two cells import various useful packages. The first cell uses the built-in magics to import the `numpy` and `matplotlib` packages.

§ %pylab inline

§ import dataiku

§ from dataiku import pandasutils as pdu

§ import pandas as pd

The third cell reads the *customers\_stacked\_prepared* dataset into a pandas dataframe named `df`.

§ # Read the dataset as a Pandas dataframe in memory

§ # Note: here, we only read the first 100K rows. Other sampling options are available

§ dataset\_customers\_stacked\_prepared = dataiku.Dataset("customers\_stacked\_prepared")

§ df = dataset\_customers\_stacked\_prepared.get\_dataframe(limit=100000)

The fourth cell audits the columns in the dataframe to provide information on the type of data in the column and the number of unique and missing values.

§ # Get some simple descriptive statistics

§ pdu.audit(df)

* Run each of the cells in order to produce the output.

Note

The notebook also now has the dataframe `df` ready in memory.

As in the Python recipe, Dataiku provides code samples to help us get started.

* Open up the list of code samples and search for `Get statistics (distribution)`.

* Copy the first code sample `my\_df["col\_0"].value\_counts()` and paste it into a new cell in the notebook.

* Edit the code to be relevant to our data and then run it:

§ df["campaign"].value\_counts()

The cell returns the number of customers who are part of the marketing campaign and the number who are not. Now we’d like to visualize the effect of *campaign* on the total amount a customer has spent.

* Since that information is in the *orders\_by\_customer* dataset, read that dataset into a new dataframe:

§ dataset\_orders = dataiku.Dataset("orders\_by\_customer")

§ df\_orders = dataset\_orders.get\_dataframe(limit=100000)

Then join it with the `df` dataframe.

* Search the code samples for `Join dataframes on columns`.

* Copy the code for *On columns with different names* to a new notebook cell, and modify it to apply to our data.

§ df\_joined = df.merge(df\_orders,left\_on="customerID",right\_on="customer\_id")

Finally, the following code produces a paneled histogram with the bar heights normalized so that it’s easier to compare across values of *campaign*.

§ pd.pivot\_table(df\_joined.reset\_index(),

§ index='index',

§ columns='campaign',

§ values=['total']).plot.hist(density=True,subplots=True)

Recall that the notebook is a lab environment, intended for experimentation. Therefore, the join we performed between the dataframes isn’t reflected in the current Flow as part of our data pipeline until we create a recipe.

* From within the notebook, click **Create Recipe > Python recipe**.

It has automatically included the *customers\_stacked\_prepared* dataset as an input, but now we’ll want to add one input and name the output.

* Add *orders\_by\_customer* as an input dataset.

* Create a new output dataset called `customers\_enriched`.

Note

In the resulting recipe, note that Dataiku adds a number of commented lines, each of which shows the beginning of a notebook cell. This way, if we need to edit the recipe in a notebook again, our existing cells are maintained.

Dataiku has added lines for the recipe output to the *customers\_enriched* dataset. However, it cannot know which dataframe we want to output. Accordingly:

* In the last line, change `pandas\_dataframe` to `df\_joined`.

* Run the recipe and see how the Flow is affected.

## What’s Next?[¶](https://knowledge.dataiku.com/latest/courses/advanced-code/python/basics.html#what-s-next "Permalink to this headline")

Congratulations! You’ve taken the first steps on Python integration in Dataiku. As you progress, you’ll find that:

* **Python is everywhere.** You can use Python code to create:

+ Bokeh webapps

+ Standard webapps with custom Python backends

+ Custom models in the Visual ML interface

+ Custom automation scenarios, metrics, and checks

+ Scripts that use the Python API to remotely control Dataiku

+ … and more!

* **Python is extensible.** You can create:

+ Code environments to manage package dependencies and versions for your projects

+ Custom Python libraries: reuse code all over the place. Should connect in to the Git-based dev workflow

+ Plugins
