# Image Classification with Code / Deep Learning for Images[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#image-classification-with-code-deep-learning-for-images "Permalink to this headline")

Deep learning models are powerful tools for image classification. In Dataiku you can build a convolutional neural network model for image classification.

However, such models can be difficult and expensive to create from scratch, especially if you don’t have a large number of images for training the model. Dataiku provides a plugin that supplies a number of pre-trained deep learning models that you can use to classify images. You can also retrain a model to specialize it on a particular set of images, a process known as transfer learning.

## Objectives[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#objectives "Permalink to this headline")

We’ll walk through how to build a convolutional network for image classification, using Keras code in Dataiku’s Visual Machine Learning.

We’ll then describe how to perform transfer learning, retraining a model on a particular set of images.

## Prerequisites[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#prerequisites "Permalink to this headline")

You should have some experience with Deep Learning with Code in Dataiku.

You should have some familiarity with Keras.

You will need access to a code environment with the necessary libraries. When creating a code environment, you can add sets of packages on the **Packages to Install** tab. Choose the Visual Deep Learning package set that corresponds to the hardware you’re running on.

In the Transfer Learning example, you will need the Deep Learning for images plugin. Install this plugin from the store.

## Recognizing Handwritten Digits[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#recognizing-handwritten-digits "Permalink to this headline")

To demonstrate how to build a model from scratch, we’ll use the MNIST database. The goal is for the model to be able to identify a handwritten number.

### Preparing the Data[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#preparing-the-data "Permalink to this headline")

Download the PNG version of the database. Create a new project. Create a new folder *mnist\_png* in the flow and populate it with the *mnist\_png.tar.gz* file.

Tip

You can also use the Download recipe to create the folder and populate it with the contents of the archive. To do this, go to the Flow, and click the **+Recipe** button. Select **Visual** > **Download**. When configuring the recipe, select **+Add a First Source** and type the URL `https://github.com/myleott/mnist\_png/raw/master/mnist\_png.tar.gz`.

In order to train and test a deep learning model for these images, we need to create train and test datasets that contain the path to each image (so that the model can find it) and the label that identifies the digit each image represents.

As a first step, we’ll create a Python recipe that uses the *mnist\_png* folder as an input and a new dataset *mnist* as the output. The code of the recipe uses the Dataiku Python API to retrieve the list of paths of all images in the folder (and its subfolders) and writes the paths as a column in the output dataset.

Note

In order to use this code in your project, you’ll need to change “9672PoPB” to the identifier for the folder in your project.

§ # -\*- coding: utf-8 -\*-

§ import dataiku

§ import pandas as pd, numpy as np

§ from dataiku import pandasutils as pdu

§ # Read recipe inputs

§ folder = dataiku.Folder("9672PoPB")

§ # Initialize data frame

§ df = pd.DataFrame(columns=['path'])

§ # Populate dataframe with paths

§ df['path'] = folder.list\_paths\_in\_partition()

§ # Write recipe outputs

§ mnist = dataiku.Dataset("mnist")

§ mnist.write\_with\_schema(df)

Our next step is to use a Prepare recipe to extract the label from the image path and whether the image belongs to the train or test sample.

Next, we’ll use a Split recipe to split the records into the train and test datasets. Now the data is ready to build a deep learning model for image classification.

### The Deep Learning Model[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#the-deep-learning-model "Permalink to this headline")

Create a Visual Analysis for the training dataset (from the dataset’s Actions menu, Lab > Visual Analysis). In the Script tab of the Visual Analysis, change the Design Sample to ensure that all 10 digits are represented in the sample. Since the training sample is only 60,000 records, you can simply choose the first 60,000 records. Save and refresh the Design Sample.

Next, create a new model with:

* **Prediction** as the task,

* *label* as the target variable

* **Deep learning** as the Expert mode, then click **Create**

This creates a new machine learning task and opens the Design tab for the task. On the Target panel, Dataiku DSS may identify this as a Regression type of ML task because *label* is a numeric column with many unique values. Change the prediction type to Multiclass classification.

Dataiku may display a message letting you know that no input feature is selected. We’ll resolve this error by configuring feature handling.

#### Features Handling[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#features-handling "Permalink to this headline")

On the Features Handling panel, turn on *path* as an input, and select **Image** as its variable type.

Select the folder that contains the image archive. *IMPORTANT:* the trained model will look for images in this directory. If we want to score new handwritten digits, they will need to be placed in this folder.

We won’t use the default code, so just remove all the code. Then, click on {} Code Samples on the top right and select the **Default Keras preprocessing for image** code sample.

Insert the Keras code then change the resized width and height variables from `197` to `28`.

#### Deep Learning Architecture[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#deep-learning-architecture "Permalink to this headline")

We now have to create our network architecture in the `build\_model()` function. We won’t use the default architecture, so just remove all the code. Then, click on **{} Code Samples** on the top right and search for “images”. Select the **CNN** architecture for image classification.

Insert the CNN code then click on **Display inputs** on the top left. You should see that the “main” feature is empty because we are only using the image data, which is in the input path\_preprocessed.

In order to build the model, we need to make a few changes to the code.

* In the line that defines `image\_shape` change `197, 197, 3` to `28, 28, 3`

* In the line that defines `image\_input\_name`, change `name\_of\_your\_image\_input\_preprocessed` to `path\_preprocessed`.

* The code sample defines a fairly complex CNN with several hidden layers and a large number of nodes within each layer. Unless you have access to a GPU, erase all of the code in the `build\_model()` function between the comment `DEFINING THE ARCHITECTURE` and the `return model` line, and replace it with the following:

§ x = Conv2D(32, kernel\_size=3, padding='same', activation='relu')(image\_input)

§ x = Conv2D(32, kernel\_size=3, padding='same', activation='relu')(x)

§ x = MaxPooling2D(pool\_size=(2, 2))(x)

§ x = Flatten()(x)

§ x = Dense(128, activation='relu')(x)

§ x = Dropout(0.2)(x)

§ predictions = Dense(n\_classes, activation='softmax')(x)

§ model = Model(inputs=image\_input, outputs=predictions)

This should be sufficient to create a good model for the images that won’t take all night to build on a laptop.

### Model Results[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#model-results "Permalink to this headline")

Click **Train** and, when complete, deploy the model to the flow, create an evaluation recipe from the model, and evaluate on the test data. In our example, the model has an accuracy of about 98.7%. Your results will vary from this example.

## Transfer Learning[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#transfer-learning "Permalink to this headline")

The MNIST dataset is useful for learning how to train a convolutional network from scratch. However, for most image-related tasks, training a model from scratch takes a very long time, and will have weak performance unless you have a sufficiently large image archive.

We will now start with a pre-trained model that has learned another dataset such as ImageNet, and use a relatively small archive of 4000 images to perform transfer learning so that the retrained model can classify cats and dogs.

### Preparing the Data[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#id1 "Permalink to this headline")

Use the same project we used with the MNIST data or create a new one. Create a new folder *cats\_dogs* in the flow and populate it with the uncompressed contents of the cats\_dogs archive.

In order to perform transfer learning for these images, we need to

Create train and test datasets that contain the path to each image (so that the model can find it) and the label that identifies the digit each image represents.

Download the specifications of a pre-trained model that will be retrained using these images.

As a first step, we’ll create a Python recipe that uses the *cats\_dogs* folder as an input and a new dataset *cats\_dogs\_labels* as the output. The code of the recipe uses the Dataiku Python API to retrieve the list of paths of all images in the folder (and its subfolders), extracts the paths, randomizes the order of records, and writes the paths as a column in the output dataset.

Randomizing the order of records ensures that each iterative training batch contains a mix of cats and dogs, and better performance than if all the cats and then all the dogs are used.

Note

In order to use this code in your project, you’ll need to change “EQysY5vS” to the identifier for the folder in your project.

§ # -\*- coding: utf-8 -\*-

§ import dataiku

§ import pandas as pd, numpy as np

§ from dataiku import pandasutils as pdu

§ # Read recipe inputs

§ folder = dataiku.Folder("EQysY5vS")

§ # Initialize dataframe

§ df = pd.DataFrame(columns=['path'])

§ # Populate dataframe with paths and labels

§ df['path'] = folder.list\_paths\_in\_partition()

§ # Randomize order of records

§ df = df.sample(frac=1).reset\_index(drop=True)

§ # Write recipe outputs

§ cats\_dogs\_labels = dataiku.Dataset("cats\_dogs\_labels")

§ cats\_dogs\_labels.write\_with\_schema(df)

Our next step is to use a Prepare recipe to extract the label from the image path.

Next, we’ll use a Split recipe to assign records into the train and test datasets.

Note

The quality of transfer learning for image classification is dependent upon the training & testing sets. In particular, different random splits of this set of 4000 images can give excellent or poor results. This underscores the importance of having a testing set, and may also show that 4000 images is not very many to train on for this application.

Finally, before going to the Lab, we need to download and place the pre-trained weights in a folder. Assuming you installed the Deep Learning for images plugin as noted in the Prerequisites, go the macro page of your project, click on the box **Download pre-trained model**, type `xception\_weights` as the output folder, and select the Xception architecture.

### The Deep Learning Model[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#id3 "Permalink to this headline")

In a Visual Analysis for the training dataset (from the dataset’s Actions menu, Lab > Visual Analysis), create a new model with:

* **Prediction** as the task,

* *label* as the target variable

* **Deep learning** as the Expert mode, then click **Create**

This creates a new machine learning task and opens the Design tab for the task. On the Target panel, verify that Dataiku DSS correctly identifies this as a Two-class classification type of ML task.

#### Features Handling[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#id4 "Permalink to this headline")

On the Features Handling panel, turn on *path* as an input, and select **Image** as its variable type.

Select the folder that contains the image archive. *IMPORTANT:* the trained model will look for images in this directory. If we want to score new images of cats and dogs, they will need to be placed in this folder.

We won’t use the default code, so just remove all the code. Then, click on {} Code Samples on the top right and select the **Default Keras preprocessing for image** code sample.

Insert the Keras code then change the resized width and height variables from `197` to `299`.

#### Deep Learning Architecture[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#id5 "Permalink to this headline")

We now have to create our network architecture in the `build\_model()` function. We won’t use the default architecture, so just remove all the code. Then, click on **{} Code Samples** on the top right and search for “images”. Open the **Pre-trained architecture to classify images (Xception)**, and scroll down to choose the **Architecture including weights download** option.

Insert the code; in order to retrain the model, we need to make a few changes to the code.

* In the line that defines `image\_shape` change `197, 197, 3` to `299, 299, 3`

* In the line that defines `image\_input\_name`, change `name\_of\_your\_image\_input\_preprocessed` to `path\_preprocessed`.

* In the line that defines `folder`, change `name\_of\_folder\_containing\_xception\_weights` to `xception\_weights`.

* Add the following lines after the call to `base\_model = Xception(include\_top=False,input\_tensor=image\_input)`. This preserves the Xception layers because they are already modified to distinguish visual features. Sometimes we will want to retrain these layers, but not now.

§ for layer in base\_model.layers:

§ layer.trainable = False

#### Deep Learning Training Settings[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#deep-learning-training-settings "Permalink to this headline")

Now that our architecture is set up, go the **Training** panel, then click on **Advanced mode** in the upper-right corner. In this code editor we will add image augmentation and a callback.

* First, replace the whole existing code with the content of the **Augmentation of images** sample.

* In the added `build\_sequences()` function, change the batch size from `8` to `16`, and replace the image input name with `path\_preprocessed`.

`ImageDataGenerator` defines how some images fed to the neural network will be randomly altered. Those alterations are subtle enough so we can distinguish its content, and their goal is to help the model generalize to unseen pictures.

`DataAugmentationSequence` defines the augmented training sequence, and takes as inputs the existing training sequence, the input containing the images (in this case, `path\_preprocessed`), the augmentations to be performed that we just defined, and the number of augmentations to perform. The number of augmentations corresponds to the number of times an image is augmented in the same batch. When in doubt, leave this parameter set to 1.

We’ll then add a new callback, a small class that can apply an operation on the model at each batch or at each epoch. You can choose one from the Keras API or code your own. This callback will reduce the learning rate as model performance on the validation set stops improving.

* Add the whole **Decrease the learning rate when no improvement in performance (Callback)** code sample to the code snippet.

### Model Results[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#id6 "Permalink to this headline")

Click **Train**. If you’re running on CPU, it will take some time to complete. When the training finishes, deploy the model to the flow, create an evaluation recipe from the model, and evaluate on the test data. In our example, you can see that the model has an accuracy of about 98.25%. Your results will vary from this example.

## Troubleshooting[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#troubleshooting "Permalink to this headline")

**Cannot reproduce the stetps.** To reproduce the steps, you’ll need Dataiku version 11.0 or above.

**Poor results.** When the amount of training data is low, the model can fail to converge, leading to varying model accuracy.

**Training is slow.** The training/scoring of Keras models can be run on either a CPU, or one or more GPUs. Training on GPUs is usually much faster, especially when images are involved. When running on a CPU, try a class rebalance sampling method, setting the number of records to 2000 or less in the **Train / Test Set** step of the **Design**.

**Code environment errors.** **Code environment errors.** Ensure the runtime environment setting in the model design is set to your deep learning code environment.

## Wrap Up[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/image-classification-code/index.html#wrap-up "Permalink to this headline")

* See a completed version of this project on the Dataiku gallery.

* See the Dataiku DSS reference documentation on deep learning.
