 _Note:_ Please right-click and open screenshot image in new tab, if it is too small to read in your current browser settings.
 
# Data Cleaning
![step_data_cleaning.png](FrtZ9u2kLNtg)

**1. Pivoting:  sensor_data dataset is pivoted to convert from [long to wide format ](https://knowledge.dataiku.com/latest/ml-analytics/time-series/concept-data-types-formats.html#time-series-data-formats)using sensor_id and timestamp.** 
![step_data_cleaning_1_pivoting.png](nMGvb8lRrqUt)

**2. Cleaning: sensor_data dataset is cleaned by parsing the timestamp columns to date format and removing outliers.**

 _Parsing:_ 
The two datasets contain different timestamp formats. The sensor_dataset has timestamps in the format yyyy-MM-dd HH:mm:ss. Using the Smart Date works great to find the correct date format of the timestamp column.
Do the parsing in-place and select Datetime with TZ as output format so that the datasets can be combined at a later stage.
![step_data_cleaning_2_parsing.png](6Mkg9ulWOzrQ)

_Removing outliers:_
As often happens, the sensor data contains some outliers resulting from incorrect measurements made by the sensors. These can be easily removed using Dataiku. As shown in the GIF below, values outside of the 5th interquartile range of each column are removed.
![outlier_removal.gif](zXb2HoxBqMaw)
![step_data_cleaning_2_outliers.png](ydeN6FqbxlFw)

 **3. Cleaning: batch_data dataset is cleaned by parsing and removing rows with missing values.** 
 
 _Parsing the start_time and end_time columns to date format:_ 
The two datasets contain different timestamp formats. The batch_dataset has timestamps in the format dd/MM/yyyy HH:mm:ss. Using the Smart Date works great to find the correct date format of the dates columns.
Do the parsing in-place and select Datetime with TZ as output format so that the datasets can be combined at a later stage.
![step_data_cleaning_3_parsing.png](tBm5xdwRIyLM)

 _Removing rows with missing recipe:_ 
Using the remove step, rows in which the batch recipe is not specified are removed. In this project we see that 2 rows are removed.
![step_data_cleaning_3_missing_recipe.png](97OJ1ByzkCkN)

 _Removing rows with missing timestamps:_ 
The same is done for batches missing a starting or ending timestamp.
![step_data_cleaning_3_missing_timestamp.png](2u3aVPdCx4MJ)

 **4. Distinct: duplicate rows are removed.** 
![step_data_cleaning_4_duplicates.png](KlAEr5zbCMfq)
 