# Active Learning for object detection problems[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#active-learning-for-object-detection-problems "Permalink to this headline")

Note

This tutorial describes how to create a project for labeling objects on images using active learning from scratch. For pre-configured solution check Active Learning for object detection problems using Dataiku Apps.

## Prerequisites[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#prerequisites "Permalink to this headline")

* You should be familiar with the basics of machine learning in Dataiku DSS.

## Technical Requirements[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#technical-requirements "Permalink to this headline")

* Access to a DSS instance of version higher than 7.0 where the ML-assisted Labeling and Object detection plugin plugins are installed.

## Setting Up[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#setting-up "Permalink to this headline")

Suppose you need to detect the location of different playing cards on the images. We’re going to download a set of unlabeled images and label them manually using active learning in a specific webapp.

### Supporting Data[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#supporting-data "Permalink to this headline")

We will use a set of images of playing cards. In total (train + test sets) it contains 363 images of playing cards of 6 types: nine, ten, jack, queen, king and ace.

We’re going to use both train and the test set images together since we’re going to ignore the test set labels and label the data manually.

### Create the Project and Set the Code Environment[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#create-the-project-and-set-the-code-environment "Permalink to this headline")

Create a new project and within the new project go to **… > Settings > Code env selection**.

* Deselect **Use DSS builtin Python env** and select an environment that has Python3 support.

* Click **Save**.

### Prepare the Data[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#prepare-the-data "Permalink to this headline")

* Create a new managed folder and call it **images**

* Upload the pictures from the train and test directories into the newly created managed folder

### Label the data[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#label-the-data "Permalink to this headline")

Now that you have 363 unlabeled images before you can start training a deep learning model to detect cards you first have to label them. To do so go to “Webapps” -> “New webapp”, select “Image object labeling” and create it. Select the **images** folder in the “Images to label” field. Enter the 6 categories, create labels and metadata datasets.

The labels dataset will contain information about objects labeled in the images. Metadata will contain additional information such as the name of the labeler, comments and date.

For now you can leave the Active learning specific part untouched.

Click “Save” and navigate to the “View” tab.

In order to start labeling select a category by clicking on it or pressing a corresponding hotkey. Then draw a rectangle around an object on the image. If the category needs to be changed, select a bounding box and then choose the right category from a drop down list on the right. Once all of the objects are selected on the image you can proceed to the next one by using the most convenient option:

* pressing the right arrow key

* clicking on “Save and next” button

* pressing a space key

Label a few samples, make sure that you have at least one label per category (grey progress bar under a category button). Once you have enough labeled samples you can start training your model.

### Training the model[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#training-the-model "Permalink to this headline")

In order to train a model we’re going to use an Object detection plugin. First thing to do is to download some related files into a **coco\_weights** managed folder using a **Download pre-trained detection model** macro:

* `weights.h5` - Keras retinanet model pretrained on the full COCO dataset

* `labels.json` - original COCO dataset labels

Now you can create a visual recipe of type “Object detection in images”, select the “Retrain object detection model” option

Fill in the following parameters:

* Folder containing the images - **images**

* Dataset of bounding boxes - **labels**

* Initial weights - **coco\_weights**

For the trained model create a new folder called **trained\_model**

In the recipe settings click on the “Label stored as object” checkbox and fill the parameters:

* Image filename column - **path**

* Label column - **label**

Note

If you have GPUs available and you’ve installed a GPU version of the Object detection plugin, you may choose to check the “Use GPU” option in the bottom of the settings

This is what your flow should look like by now:

## Putting Active Learning in Place[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#putting-active-learning-in-place "Permalink to this headline")

We would like to improve the model performance by labeling more rows and adding them to the training set. This is where active learning comes into play. We will:

* Set up the recipe that uses the retinanet model to determine the order in which rows should be manually labeled.

* Configure the labeling webapp to use a queries dataset.

* Set up a scenario that automates the rebuilding of the ML model and then updates the order in which rows should be manually labeled.

### Set Up the Query Sampler Recipe[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#set-up-the-query-sampler-recipe "Permalink to this headline")

* Click **+ Recipe > ML-assisted Labeling > Object detection query sampler**.

* For the Managed folder with model weights, select *trained\_model*.

* For the Unlabeled Data, select *images*.

* For the output Data to be labeled, create a new dataset called `queries`.

* Click **Create**.

Ensure that **Smallest confidence sampling** is selected as the Query strategy. **Run** the recipe.

### Configure Labeling Application to use the queries dataset[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#configure-labeling-application-to-use-the-queries-dataset "Permalink to this headline")

Start by creating a web application to label images:

* Go to webapp that you’ve created on the previous steps

* Select the queries dataset in the settings under the **ACTIVE LEARNING SPECIFIC** section

* Save the settings, the webapp will reload.

Now, if you open the View tab again you will see that the active learning is now enabled, thanks to the generated queries dataset.

The goal of active learning is to provide a labeler first with samples that the model had most difficulties with. There’s a widget with a color scale showing how efficient current sampling is compared to random. If the tick is on the right side of the widget it means that the model should be retrained and the queries dataset needs to be regenerated.

From here on the iterative labeling process starts:

* Label few samples

* Retrain the model

* Regenerate the queries

* Restart the webapp

This process may be automated by a scenario.

### Set Up the Training Scenario[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#set-up-the-training-scenario "Permalink to this headline")

The Flow is now complete; however, the model must be manually retrained in order to generate new queries. The active learning plugin provides a scenario trigger to do it automatically. Let’s set it up.

* From the **Jobs** dropdown in the top navigation bar, select **Scenarios**.

* Click **+ New Scenario**

* Name it `Retrain cards detection` and click **Create**.

* Click **Add Trigger > ML-assisted Labeling > Trigger every n labeling**.

+ Name the trigger `Every 10 labeling`.

+ Set Run every (seconds) to `20`.

+ Set Labeling count to `10`.

+ Set the Metadata dataset to *metadata*.

+ Set the Queries dataset to *queries*.

* In the **Steps** tab:

+ Click **Add Step > Build/Train**.

- Name it `Rebuild model`.

- Click **Add Folder to Build** and add *trained\_model*

- For the Build mode, select **Build only this dataset**.

+ Click **Add Step > Build/Train**.

- Name it `Queries`.

- Click **Add Dataset to Build** and add *queries*.

- For the Build mode, select **Build only this dataset**.

+ Click **Add Step > Restart webapp**.

- Name it `Detect cards`.

- Select *detect cards* as the web app to restart.

In the scenario **Settings** tab, set the auto-trigger of the scenario on. What happens now is that the scenario will trigger the generation of new queries every time 10 samples are labeled.

## Label data[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#label-data "Permalink to this headline")

This is it! You can use the webapp by going to its **View** tab. Start labeling cards. Notice that you can use keystrokes to select categories faster. Simply hit **j** or **n**, on your keyboard to select the category and draw the bounding box of the corresponding type. You may also hit Space or right arrow button to confirm your labels and proceed to a next sample.

As you label, you will see notifications pop every 10 labels.

## What’s Next[¶](https://knowledge.dataiku.com/latest/kb/analytics-ml/active-learning/active-learning-object-detection.html#what-s-next "Permalink to this headline")

For more on active learning, see the following posts on *Data From the Trenches*:

* A Proactive Look at Active Learning Packages

* Diverse Mini-Batch Active Learning: A Reproduction Exercise

References
