{
  "metadata": {
    "hide_input": false,
    "kernelspec": {
      "name": "py-dku-venv-py_39_mlflow_in_dku",
      "display_name": "Python (env py_39_mlflow_in_dku)",
      "language": "python"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.20",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "createdOn": 1700068576864,
    "tags": [],
    "customFields": {},
    "creator": "camille.cochener",
    "modifiedBy": "camille.cochener"
  },
  "nbformat": 4,
  "nbformat_minor": 1,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Experiment Tracking for time series multi output model with Keras\n\nIn this notebook, you can:\n\n1. Train Keras time series models to predict multiple numerical targets\n2. Log those models in the MLflow format so that they can be compared using the DSS [Experiment Tracking interface](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/viewing.html?_gl\u003d1*shnz4d*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..)\n\nAlthough MLflow provides the [mlflow.sklearn.log_model](https://www.mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.log_model) function to log models, you will rely on the more general [mlflow.pyfunc.PythonModel](https://www.mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.PythonModel) module to enjoy greater flexibility. If needed, please consult our [pyfunc tutorial](https://developer.dataiku.com/latest/tutorials/machine-learning/experiment-tracking/xgboost-pyfunc/index.html) to get familiar with that module."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Preparing the experiment\n\nThe first step is to prepare the ground for the experiment tracking:\n\nThe following code imports all libraries and defines constant variables, handles and function necessary to the training and tracking of Keras models. \n\nMake sure to change the ids and names to your own parameters.\n\nFor more information regarding experiment tracking in code, refer to our [documentation](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/tracking.html?_gl\u003d1*124ki6d*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..)."
      ]
    },
    {
      "execution_count": 44,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# General libraries\nimport os\nimport shutil\nimport dataiku\nimport itertools\nimport numpy as np\nimport pandas as pd\nfrom dataiku import pandasutils as pdu\nfrom dataikuapi.dss.ml import DSSPredictionMLTaskSettings\n\n# ML\nimport tensorflow as tf\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.optimizers import Adam\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\n# MLflow\nimport mlflow\nimport mlflow.pytorch"
      ],
      "outputs": []
    },
    {
      "execution_count": 12,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Replace these constants with your own values\nPREDICTION_TYPE \u003d \"OTHER\"\nEXPERIMENT_FOLDER_ID \u003d \"eXga6z7K\"      \nEXPERIMENT_NAME \u003d \"Keras-MO-TS-Model\"             \nMLFLOW_CODE_ENV_NAME \u003d \"py_39_mlflow_in_dku\"        \nSAVED_MODEL_NAME \u003d \"Keras Multi-Output Time Series Model\"\nDEPLOYMENT_METRIC \u003d \"test_loss\"\nEVAL_DATASET \u003d \"validation_ts\"\nTARGETS \u003d [\"output1\", \"output2\"]\nAUTO_DEPLOY \u003d False\n\n# Dictionary with path to save intermediary model\nartifacts \u003d {\n    SAVED_MODEL_NAME: \"./keras_model_multioutput_ts.pth\"\n}"
      ],
      "outputs": []
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Get a handle on a Managed Folder to store the experiments\nproject \u003d dataiku.api_client().get_default_project()\nmf \u003d project.get_managed_folder(EXPERIMENT_FOLDER_ID)\n\n# Create a mlflow_extension object to easily collect information for the promotion step\nmlflow_extension \u003d project.get_mlflow_extension()"
      ],
      "outputs": []
    },
    {
      "execution_count": 13,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# (Opt) Garbage collect experiments - this will delete experiments that were removed\nmlflow_extension.garbage_collect()"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Loading the data\n\nThe second step to training time series models is to obtain the data.\n\nHere, the datasets are loaded from the Flow using the DSS Python API:\n- A `train_ts` dataset will be used to train the time series models\n- A `validation_ts` dataset will be used to assess the performance of the time series models"
      ]
    },
    {
      "execution_count": 19,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Load your data into a Pandas DataFrame\ntrain \u003d dataiku.Dataset(\"train_ts\")\ndf_train \u003d train.get_dataframe()\n\nvalidation \u003d dataiku.Dataset(\"validation_ts\")\ndf_validation \u003d validation.get_dataframe()\n\n# Split the data into features and target\nX_train \u003d df_train.drop(TARGETS, axis\u003d1)\ny_train \u003d df_train[TARGETS]\n\nX_test \u003d df_validation.drop(TARGETS, axis\u003d1)\ny_test \u003d df_validation[TARGETS]\n\nX_train \u003d X_train.drop(\"datetime\", axis\u003d1)\nX_test \u003d X_test.drop(\"datetime\", axis\u003d1)"
      ],
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/opt/dataiku/code-env/lib/python3.8/site-packages/pandas/core/indexes/base.py:395: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\nDeprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n  elif issubclass(data.dtype.type, np.bool) or is_bool_dtype(data):\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Preprocessing (Optional)"
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Standardize the data\n#scaler \u003d StandardScaler()\n#X_train_scaled \u003d scaler.fit_transform(X_train)\n#X_test_scaled \u003d scaler.transform(X_test)"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Model Training and hyperparameter grid\n\nNow define a function containing the architecture of the neural network as well as the hyperparameter grid to go through."
      ]
    },
    {
      "execution_count": 36,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define the neural network model\ndef create_model(learning_rate):\n    model \u003d Sequential()\n    model.add(Dense(64, input_shape\u003d(X_train.shape[1],), activation\u003d\u0027relu\u0027))\n    model.add(Dense(32, activation\u003d\u0027relu\u0027))\n    model.add(Dense(2, activation\u003d\u0027linear\u0027))\n\n    # Compile the model\n    model.compile(optimizer\u003dAdam(learning_rate\u003dlearning_rate), loss\u003d\u0027mean_squared_error\u0027)\n    \n    return model"
      ],
      "outputs": []
    },
    {
      "execution_count": 37,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Parameter grid\nparameter_grid \u003d {\n    \u0027batch_size\u0027: [32], \n    \u0027learning_rate\u0027: [0.001],\n    \u0027num_epochs\u0027: [50]\n}"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Experiment runs\n\nAt last, the following piece of code runs training while tracking all the experiments. After a successful run, you will be able to deploy the model either visually or programmatically.\n\nThe following code can be split into the different steps:\n\n1. Create an `mlflow` context using the `setup_mlflow()` method from the Dataiku API.\n2. Create an experiment\n3. Loop through the list of hyperparameter combinations so that for each combination, you start a run in which you:\n - Create and train a model.\n - Collect trained model metrics on test set\n - Serialize + save the model. This is a necessary intermediary step.\n - Wrap that model in a `PytorchModelWrapper` so that the model can be logged as an MLflow python function model.\n - Log the wrapper along with the collected metrics and other model metadata (hyperparemeters, epochs, code environment…)."
      ]
    },
    {
      "execution_count": 45,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "with project.setup_mlflow(mf) as mlflow: \n    \n    # If the experiment did not already exist, we create it\n    if len(mlflow_extension.list_experiments()) \u003e\u003d 1:\n        if EXPERIMENT_NAME in [exp[\u0027name\u0027] for exp in mlflow_extension.list_experiments()[\u0027experiments\u0027]]:\n            print(\"Experiment already existed with name : {}\".format(EXPERIMENT_NAME))\n        else:\n            experiment_id \u003d mlflow.create_experiment(EXPERIMENT_NAME)\n            print(\"Experiment created with name : {}\".format(EXPERIMENT_NAME))\n    else:\n        experiment_id \u003d mlflow.create_experiment(EXPERIMENT_NAME)\n        print(\"Experiment created with name : {}\".format(EXPERIMENT_NAME))\n        \n    # For display\n    experiment \u003d mlflow.get_experiment_by_name(EXPERIMENT_NAME)\n    experiment_id \u003d experiment.experiment_id\n    print(\"Experiment name is: {} and ID is: {} \".format(EXPERIMENT_NAME, experiment_id))\n    \n    # Define the Python wrapper\n    class KerasModelWrapper(mlflow.pyfunc.PythonModel):\n        def load_context(self, context):\n            print(context.artifacts[SAVED_MODEL_NAME])\n            self.model \u003d tf.keras.models.load_model(context.artifacts[SAVED_MODEL_NAME])\n\n        def predict(self, context, model_input):\n            model_input \u003d model_input.drop(TARGETS + [\"datetime\"], axis\u003d1)\n            return self.model.predict(model_input)\n\n    for params in itertools.product(*parameter_grid.values()):\n        # Unpack the parameters\n        batch_size, learning_rate, num_epochs \u003d params\n        \n        with mlflow.start_run(experiment_id\u003dexperiment_id) as run:\n            # Display run ID\n            run_id \u003d run.info.run_id\n            print(f\u0027Starting run {run_id} ...\\n{params}\u0027)\n            \n            # Log hyperparameters\n            mlflow.log_param(\"batch_size\", batch_size)\n            mlflow.log_param(\"learning_rate\", learning_rate)\n            mlflow.log_param(\"num_epochs\", num_epochs)\n            \n            model \u003d create_model(learning_rate)\n            print(model.summary())\n            \n            # Train the model\n            model.fit(X_train, y_train, epochs\u003dnum_epochs, batch_size\u003dbatch_size, validation_split\u003d0.1)\n                        \n            # Evaluate the model on the test set\n            loss \u003d model.evaluate(X_test, y_test)\n            print(f\u0027Test Loss: {loss}\u0027)\n            \n            # Log test loss\n            mlflow.log_metric(\"test_loss\", loss)\n            \n            # Serialize and save the model\n            if os.path.exists(artifacts.get(SAVED_MODEL_NAME)):\n                shutil.rmtree(artifacts.get(SAVED_MODEL_NAME))\n            model.save(artifacts.get(SAVED_MODEL_NAME))\n            \n            mlflow_pyfunc_model_path \u003d f\"{type(model).__name__}-{run.info.run_id}\"\n            \n            # Save the PyTorch model\n            mlflow.pyfunc.log_model(\n                artifact_path\u003dmlflow_pyfunc_model_path, python_model\u003dKerasModelWrapper(),\n                artifacts\u003dartifacts\n            )\n            \n            # Sets the type of the model, and optionally other information useful to deploy or evaluate it.\n            mlflow_extension.set_run_inference_info(run_id\u003drun_id, \n                                                    prediction_type\u003dPREDICTION_TYPE, \n                                                    code_env_name\u003dMLFLOW_CODE_ENV_NAME)\n            # This information is leveraged to filter saved models on their prediction type and prefill \n            # the classes when deploying using the GUI an MLflow model as a version of a DSS Saved Model.\n            \n            print(f\u0027Run {run.info.run_id} done\\n{\"-\"*40}\u0027)      "
      ],
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Experiment already existed with name : Keras-MO-TS-Model\nExperiment name is: Keras-MO-TS-Model and ID is: keras_mo_ts_model \nStarting run 2023_11_17T16_38_55 ...\n(32, 0.001, 50)\nModel: \"sequential_8\"\n_________________________________________________________________\n Layer (type)                Output Shape              Param #   \n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n dense_20 (Dense)            (None, 64)                256       \n                                                                 \n dense_21 (Dense)            (None, 32)                2080      \n                                                                 \n dense_22 (Dense)            (None, 2)                 66        \n                                                                 \n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nTotal params: 2,402\nTrainable params: 2,402\nNon-trainable params: 0\n_________________________________________________________________\nNone\nEpoch 1/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 9ms/step - loss: 1.1526 - val_loss: 0.9009\nEpoch 2/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.8499 - val_loss: 0.6143\nEpoch 3/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.5619 - val_loss: 0.3908\nEpoch 4/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.3769 - val_loss: 0.2588\nEpoch 5/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 5ms/step - loss: 0.2854 - val_loss: 0.2179\nEpoch 6/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2580 - val_loss: 0.2105\nEpoch 7/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2560 - val_loss: 0.2114\nEpoch 8/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 3ms/step - loss: 0.2533 - val_loss: 0.2099\nEpoch 9/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2528 - val_loss: 0.2090\nEpoch 10/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2504 - val_loss: 0.2090\nEpoch 11/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2506 - val_loss: 0.2050\nEpoch 12/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2493 - val_loss: 0.2107\nEpoch 13/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2486 - val_loss: 0.2066\nEpoch 14/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2487 - val_loss: 0.2123\nEpoch 15/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2477 - val_loss: 0.2102\nEpoch 16/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2487 - val_loss: 0.2094\nEpoch 17/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 5ms/step - loss: 0.2480 - val_loss: 0.2069\nEpoch 18/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2465 - val_loss: 0.2057\nEpoch 19/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2458 - val_loss: 0.2098\nEpoch 20/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2488 - val_loss: 0.2045\nEpoch 21/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2457 - val_loss: 0.2080\nEpoch 22/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2445 - val_loss: 0.2089\nEpoch 23/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2451 - val_loss: 0.2065\nEpoch 24/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2446 - val_loss: 0.2114\nEpoch 25/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 5ms/step - loss: 0.2442 - val_loss: 0.2052\nEpoch 26/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2443 - val_loss: 0.2162\nEpoch 27/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2450 - val_loss: 0.2053\nEpoch 28/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2438 - val_loss: 0.2102\nEpoch 29/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2455 - val_loss: 0.2040\nEpoch 30/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2507 - val_loss: 0.2142\nEpoch 31/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2443 - val_loss: 0.2071\nEpoch 32/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2426 - val_loss: 0.2084\nEpoch 33/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2449 - val_loss: 0.2073\nEpoch 34/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2438 - val_loss: 0.2124\nEpoch 35/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2430 - val_loss: 0.2042\nEpoch 36/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2436 - val_loss: 0.2081\nEpoch 37/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2432 - val_loss: 0.2073\nEpoch 38/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2440 - val_loss: 0.2047\nEpoch 39/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2443 - val_loss: 0.2078\nEpoch 40/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 5ms/step - loss: 0.2416 - val_loss: 0.2080\nEpoch 41/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2419 - val_loss: 0.2095\nEpoch 42/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2431 - val_loss: 0.2081\nEpoch 43/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2421 - val_loss: 0.2089\nEpoch 44/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2414 - val_loss: 0.2081\nEpoch 45/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2428 - val_loss: 0.2034\nEpoch 46/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2421 - val_loss: 0.2079\nEpoch 47/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2417 - val_loss: 0.2050\nEpoch 48/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2435 - val_loss: 0.2047\nEpoch 49/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 2ms/step - loss: 0.2416 - val_loss: 0.2084\nEpoch 50/50\n20/20 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 4ms/step - loss: 0.2436 - val_loss: 0.2102\n5/5 [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d] - 0s 1ms/step - loss: 0.2423\nTest Loss: 0.2422688603401184\nINFO:tensorflow:Assets written to: ./keras_model_multioutput_ts.pth/assets\n"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "INFO:tensorflow:Assets written to: ./keras_model_multioutput_ts.pth/assets\n"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Run 2023_11_17T16_38_55 done\n----------------------------------------\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 6. Deploying the model in the Flow (Optional)\n\nYou can now deploy your model either via the [Experiment Tracking interface](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/viewing.html?_gl\u003d1*18yft7r*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..) or through our [Python API](https://doc.dataiku.com/dss/latest/mlops/mlflow-models/importing.html?_gl\u003d1*84hye6*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..). In either case, you will need an evaluation dataset that contains a `target`column and the features. This can be the `validation` set used in this notebook.\n\nIn the code below, if the `AUTO_DEPLOY` is set to `True`, the model that will be deployed is the one with the best metric, each time this notebook/recipe runs."
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "if AUTO_DEPLOY:\n    with project.setup_mlflow(mf) as mlflow: \n\n        # Get experiment\n        experiment \u003d mlflow.get_experiment_by_name(EXPERIMENT_NAME)\n        print(\"Experiment: \", experiment)\n\n        # List the runs and get the one with the best accuracy score\n        print(\"Looking for the best run within the experiment\")\n        best_run \u003d None\n\n        for index, run_info in mlflow.search_runs(experiment.experiment_id).iterrows():\n            run \u003d mlflow.get_run(run_info[\"run_id\"])\n            if best_run is None:\n                best_run \u003d mlflow.get_run(run_info[\"run_id\"])\n            elif run.data.metrics.get(DEPLOYMENT_METRIC, 0) \u003c best_run.data.metrics.get(DEPLOYMENT_METRIC, 0):\n                best_run \u003d run\n        print(f\"Run id {best_run.info.run_id} with {DEPLOYMENT_METRIC}\u003d{best_run.data.metrics.get(DEPLOYMENT_METRIC)}\")\n\n        # Deploy the model on the flow\n        run_id \u003d best_run.info.run_id\n        prediction_type \u003d DSSPredictionMLTaskSettings.PredictionTypes.OTHER\n\n        # Get or create the Saved Model\n        sm_id \u003d None\n        for sm in project.list_saved_models():\n            if sm[\"name\"] !\u003d SAVED_MODEL_NAME:\n                continue\n            else:\n                sm_id \u003d sm[\"id\"]\n                print(f\"Found Saved Model {sm[\u0027name\u0027]} with id {sm[\u0027id\u0027]}\")\n                break\n        if sm_id:\n            saved_model \u003d project.get_saved_model(sm_id)\n        else:\n            saved_model \u003d project.create_mlflow_pyfunc_model(SAVED_MODEL_NAME, PREDICTION_TYPE)\n            sm_id \u003d saved_model.id\n            print(f\"Saved Model not found, created new one with id {sm_id}\")\n\n        # Define model version\n        model_versions\u003d[model[\u0027id\u0027] for model in saved_model.list_versions()]\n\n        if model_versions is None:\n            version_id \u003d \"v1\"\n        else:\n            i\u003d1\n            print(f\"model_versions: {model_versions}\")\n            while \"v{}\".format(i) in model_versions:\n                i+\u003d1\n            version_id \u003d \"v{}\".format(i)\n\n        print(f\"Deploying the model {SAVED_MODEL_NAME} on the flow with version id {version_id}\")\n        sm_external_model_version_handler \u003d mlflow_extension.deploy_run_model(\n            run_id,\n            sm_id,\n            code_env_name\u003dMLFLOW_CODE_ENV_NAME,\n            version_id\u003dversion_id,\n        )"
      ],
      "outputs": []
    }
  ]
}