{
  "metadata": {
    "hide_input": false,
    "kernelspec": {
      "name": "py-dku-venv-py_39_mlflow_in_dku",
      "display_name": "Python (env py_39_mlflow_in_dku)",
      "language": "python"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.20",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "createdOn": 1699879287797,
    "tags": [],
    "customFields": {},
    "creator": "camille.cochener",
    "modifiedBy": "camille.cochener"
  },
  "nbformat": 4,
  "nbformat_minor": 1,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Experiment Tracking for regression with PyTorch\n\nIn this notebook, you can:\n\n1. Train PyTorch regression models to predict a numerical target\n2. Log those models in the MLflow format so that they can be compared using the DSS [Experiment Tracking interface](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/viewing.html?_gl\u003d1*shnz4d*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..)\n\nAlthough MLflow provides the [mlflow.sklearn.log_model](https://www.mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.log_model) function to log models, you will rely on the more general [mlflow.pyfunc.PythonModel](https://www.mlflow.org/docs/latest/python_api/mlflow.pyfunc.html#mlflow.pyfunc.PythonModel) module to enjoy greater flexibility. If needed, please consult our [pyfunc tutorial](https://developer.dataiku.com/latest/tutorials/machine-learning/experiment-tracking/xgboost-pyfunc/index.html) to get familiar with that module."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Preparing the experiment\n\nThe first step is to prepare the ground for the experiment tracking:\n\nThe following code imports all libraries and defines constant variables, handles and function necessary to the training and tracking of PyTorch models. \n\nMake sure to change the ids and names to your own parameters.\n\nFor more information regarding experiment tracking in code, refer to our [documentation](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/tracking.html?_gl\u003d1*124ki6d*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..)."
      ]
    },
    {
      "execution_count": 1,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# General libraries\nimport os\nimport shutil\nimport dataiku\nimport itertools\nimport pandas as pd\nfrom dataiku import pandasutils as pdu\nfrom dataikuapi.dss.ml import DSSPredictionMLTaskSettings\n\n# ML\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader, TensorDataset\nfrom sklearn.model_selection import train_test_split\n\n# MLflow\nimport mlflow\nimport mlflow.pytorch"
      ],
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/opt/dataiku/python/dataiku/container/runner.py:404: DeprecationWarning: invalid escape sequence \\s\n  access_token_purge \u003d \"\"\"\n/opt/dataiku/python/dataiku/container/runner.py:470: DeprecationWarning: invalid escape sequence \\:\n  main_location \u003d \"\"\"\n/opt/dataiku/python/dataiku/container/runner.py:617: DeprecationWarning: invalid escape sequence \\:\n  main_location \u003d \"\"\"\n/opt/dataiku/code-env/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n  from .autonotebook import tqdm as notebook_tqdm\n/opt/dataiku/code-env/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API\n  warnings.warn(\"pkg_resources is deprecated as an API\", DeprecationWarning)\n/opt/dataiku/code-env/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace(\u0027mpl_toolkits\u0027)`.\nImplementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages\n  declare_namespace(pkg)\n/opt/dataiku/code-env/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace(\u0027google\u0027)`.\nImplementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages\n  declare_namespace(pkg)\n"
        }
      ]
    },
    {
      "execution_count": 10,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Replace these constants with your own values\nPREDICTION_TYPE \u003d \"REGRESSION\"\nEXPERIMENT_FOLDER_ID \u003d \"pshnkijv\"         \nEXPERIMENT_NAME \u003d \"PyTorch-Regression-Model\"               \nMLFLOW_CODE_ENV_NAME \u003d \"py_39_mlflow_in_dku\"          \nSAVED_MODEL_NAME \u003d \"PyTorch Regression Model\"       \nDEPLOYMENT_METRIC \u003d \"test_loss\"\nEVAL_DATASET \u003d \"validation\"\nTARGET_NAME \u003d \"target\"\nAUTO_DEPLOY \u003d True\n\n# Dictionary with path to save intermediary model\nartifacts \u003d {\n    SAVED_MODEL_NAME: \"./pytorch_regression_model.pth\"\n}"
      ],
      "outputs": []
    },
    {
      "execution_count": 3,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Get a handle on a Managed Folder to store the experiments\nproject \u003d dataiku.api_client().get_default_project()\nmf \u003d project.get_managed_folder(EXPERIMENT_FOLDER_ID)\n\n# Create a mlflow_extension object to easily collect information for the promotion step\nmlflow_extension \u003d project.get_mlflow_extension()"
      ],
      "outputs": []
    },
    {
      "execution_count": 4,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# (Opt) Garbage collect experiments - this will delete experiments that were removed\nmlflow_extension.garbage_collect()"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Loading the data\n\nThe second step to training regression models is to obtain the data.\n\nHere, the datasets are loaded from the Flow using the DSS Python API:\n- A `train` dataset will be used to train the regressors\n- A `validation` dataset will be used to assess the performance of the regressors"
      ]
    },
    {
      "execution_count": 5,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Load your data into a Pandas DataFrame\ntrain \u003d dataiku.Dataset(\"train\")\ndf_train \u003d train.get_dataframe()\n\nvalidation \u003d dataiku.Dataset(\"validation\")\ndf_validation \u003d validation.get_dataframe()\n\n# Split the data into features and target\nX_train \u003d df_train.drop(\u0027target\u0027, axis\u003d1)\ny_train \u003d df_train[\u0027target\u0027]\n\nX_test \u003d df_validation.drop(\u0027target\u0027, axis\u003d1)\ny_test \u003d df_validation[\u0027target\u0027]"
      ],
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/opt/dataiku/code-env/lib/python3.8/site-packages/pandas/core/indexes/base.py:395: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\nDeprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n  elif issubclass(data.dtype.type, np.bool) or is_bool_dtype(data):\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Converting raw data to PyTorch DataLoader\n\nBefore you can start training and evaluating your PyTorch models, you will have to convert your data to 2 different DataLoaders (train and validation).\n\nThe [torch.utils.data.DataLoader](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) function will allow to create such data loader.\n\nFor more information regarding the conversion of datasets to PyTorch DataLoaders, refer to [PyTorch documentation](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#preparing-your-data-for-training-with-dataloaders). "
      ]
    },
    {
      "execution_count": 6,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert the Pandas DataFrames to PyTorch tensors\nX_train_tensor \u003d torch.from_numpy(X_train.values).float()\ny_train_tensor \u003d torch.from_numpy(y_train.values).view(-1, 1).float()\n\nX_test_tensor \u003d torch.from_numpy(X_test.values).float()\ny_test_tensor \u003d torch.from_numpy(y_test.values).view(-1, 1).float()\n\n# Create DataLoader for training and test sets\ntrain_dataset \u003d TensorDataset(X_train_tensor, y_train_tensor)\ntest_dataset \u003d TensorDataset(X_test_tensor, y_test_tensor)\n\ntrain_loader \u003d DataLoader(dataset\u003dtrain_dataset, batch_size\u003d32, shuffle\u003dTrue)\ntest_loader \u003d DataLoader(dataset\u003dtest_dataset, batch_size\u003d32, shuffle\u003dFalse)"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Model and hyperparameter grid\n\nNow define a class containing the architecture of the neural network as well as the hyperparameter grid to go through."
      ]
    },
    {
      "execution_count": 7,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define a simple neural network\nclass SimpleNN(nn.Module):\n    def __init__(self, input_size, hidden_size, output_size):\n        super(SimpleNN, self).__init__()\n        self.fc1 \u003d nn.Linear(input_size, hidden_size)\n        self.relu \u003d nn.ReLU()\n        self.fc2 \u003d nn.Linear(hidden_size, output_size)\n\n    def forward(self, x):\n        x \u003d self.fc1(x)\n        x \u003d self.relu(x)\n        x \u003d self.fc2(x)\n        return x"
      ],
      "outputs": []
    },
    {
      "execution_count": 8,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Parameter grid\nparameter_grid \u003d {\n    \u0027input_size\u0027: [X_train.shape[1]],  # Assuming input size is the number of features\n    \u0027hidden_size\u0027: [64, 128],  # Try different hidden layer sizes\n    \u0027output_size\u0027: [1],  # Assuming a regression task\n    \u0027learning_rate\u0027: [0.01],\n    \u0027num_epochs\u0027: [20]\n}"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Experiment runs\n\nAt last, the following piece of code runs training while tracking all the experiments. After a successful run, you will be able to deploy the model either visually or programmatically.\n\nThe following code can be split into the different steps:\n\n1. Create an `mlflow` context using the `setup_mlflow()` method from the Dataiku API.\n2. Create an experiment\n3. Loop through the list of hyperparameter combinations so that for each combination, you start a run in which you:\n - Create and train a model.\n - Collect trained model metrics on test set\n - Serialize + save the model. This is a necessary intermediary step.\n - Wrap that model in a `PytorchModelWrapper` so that the model can be logged as an MLflow python function model.\n - Log the wrapper along with the collected metrics and other model metadata (hyperparemeters, epochs, code environment…)."
      ]
    },
    {
      "execution_count": 9,
      "cell_type": "code",
      "metadata": {
        "scrolled": true
      },
      "source": [
        "with project.setup_mlflow(mf) as mlflow: \n    \n    # If the experiment did not already exist, we create it\n    if len(mlflow_extension.list_experiments()) \u003e\u003d 1:\n        if EXPERIMENT_NAME in [exp[\u0027name\u0027] for exp in mlflow_extension.list_experiments()[\u0027experiments\u0027]]:\n            print(\"Experiment already existed with name : {}\".format(EXPERIMENT_NAME))\n        else:\n            experiment_id \u003d mlflow.create_experiment(EXPERIMENT_NAME)\n            print(\"Experiment created with name : {}\".format(EXPERIMENT_NAME))\n    else:\n        experiment_id \u003d mlflow.create_experiment(EXPERIMENT_NAME)\n        print(\"Experiment created with name : {}\".format(EXPERIMENT_NAME))\n    \n    # For display\n    experiment \u003d mlflow.get_experiment_by_name(EXPERIMENT_NAME)\n    experiment_id \u003d experiment.experiment_id\n    print(\"Experiment name is: {} and ID is: {} \".format(EXPERIMENT_NAME, experiment_id))\n    \n    # Iterate over the parameter grid\n    for params in itertools.product(*parameter_grid.values()):\n        # Unpack the parameters\n        input_size, hidden_size, output_size, learning_rate, num_epochs \u003d params\n\n        # Start MLflow run\n        with mlflow.start_run(experiment_id\u003dexperiment_id) as run:\n            \n            # Display run ID\n            run_id \u003d run.info.run_id\n            print(f\u0027Starting run {run_id} ...\\n{params}\u0027)\n            \n            # Initialize the model, loss function, and optimizer\n            model \u003d SimpleNN(input_size, hidden_size, output_size)\n            criterion \u003d nn.MSELoss()\n            optimizer \u003d optim.Adam(model.parameters(), lr\u003dlearning_rate)\n            \n            # Log hyperparameters\n            mlflow.log_param(\"input_size\", input_size)\n            mlflow.log_param(\"hidden_size\", hidden_size)\n            mlflow.log_param(\"output_size\", output_size)\n            mlflow.log_param(\"learning_rate\", learning_rate)\n            mlflow.log_param(\"num_epochs\", num_epochs)\n\n            # Training the model\n            for epoch in range(num_epochs):\n                for inputs, targets in train_loader:\n                    optimizer.zero_grad()\n                    outputs \u003d model(inputs)\n                    loss \u003d criterion(outputs, targets)\n                    loss.backward()\n                    optimizer.step()\n\n                # Log training loss for each epoch\n                mlflow.log_metric(\"train_loss\", loss.item(), step\u003depoch)\n\n                print(f\u0027Epoch {epoch+1}/{num_epochs}, Loss: {loss.item()}\u0027)\n            \n            # Define the Python wrapper\n            class PytorchModelWrapper(mlflow.pyfunc.PythonModel):\n                def load_context(self, context):\n                    print(context.artifacts[SAVED_MODEL_NAME])\n                    self.model \u003d mlflow.pytorch.load_model(context.artifacts[SAVED_MODEL_NAME])\n\n                def predict(self, context, model_input):\n                    model_input \u003d torch.FloatTensor(model_input.values.astype(\"float32\"))\n                    outputs \u003d model(model_input)\n                    outputs \u003d pd.DataFrame(outputs.detach().numpy())\n                    return outputs\n            \n            # Testing the model\n            model.eval()\n            test_loss \u003d 0.0\n            with torch.no_grad():\n                for inputs, targets in test_loader:\n                    outputs \u003d model(inputs)\n                    test_loss +\u003d criterion(outputs, targets).item()\n\n            avg_test_loss \u003d test_loss / len(test_loader)\n\n            # Log test loss\n            mlflow.log_metric(\"test_loss\", avg_test_loss)\n\n            print(f\u0027Average Test Loss: {avg_test_loss}\u0027)\n            \n            # Serialize and save the model\n            if os.path.exists(artifacts.get(SAVED_MODEL_NAME)):\n                shutil.rmtree(artifacts.get(SAVED_MODEL_NAME))\n            mlflow.pytorch.save_model(model, artifacts.get(SAVED_MODEL_NAME))\n            \n            mlflow_pyfunc_model_path \u003d f\"{type(model).__name__}-{run.info.run_id}\"\n            \n            # Save the PyTorch model\n            mlflow.pyfunc.log_model(\n                artifact_path\u003dmlflow_pyfunc_model_path, python_model\u003dPytorchModelWrapper(),\n                artifacts\u003dartifacts\n            )\n            \n            # Sets the type of the model, and optionally other information useful to deploy or evaluate it.\n            mlflow_extension.set_run_inference_info(run_id\u003drun_id, \n                                                    prediction_type\u003dPREDICTION_TYPE, \n                                                    code_env_name\u003dMLFLOW_CODE_ENV_NAME)\n            # This information is leveraged to filter saved models on their prediction type and prefill \n            # the classes when deploying using the GUI an MLflow model as a version of a DSS Saved Model.\n            \n            print(f\u0027Run {run.info.run_id} done\\n{\"-\"*40}\u0027)            "
      ],
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/opt/dataiku/python/dataikuapi/dss_plugin_mlflow/artifact_repository.py:17: DeprecationWarning: invalid escape sequence \\w\n  pattern \u003d re.compile(\"^(\\w+\\.)?\\w{8}\")\n"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Experiment already existed with name : PyTorch-Regression-Model\nExperiment name is: PyTorch-Regression-Model and ID is: pytorch_regression_ \nStarting run 2023_11_21T17_48_06 ...\n(10, 64, 1, 0.01, 20)\nEpoch 1/20, Loss: 13105.1865234375\nEpoch 2/20, Loss: 8008.1005859375\nEpoch 3/20, Loss: 4371.2294921875\nEpoch 4/20, Loss: 3891.434814453125\nEpoch 5/20, Loss: 1488.7242431640625\nEpoch 6/20, Loss: 683.1507568359375\nEpoch 7/20, Loss: 303.3534851074219\nEpoch 8/20, Loss: 390.7386779785156\nEpoch 9/20, Loss: 290.3238525390625\nEpoch 10/20, Loss: 263.0352478027344\nEpoch 11/20, Loss: 238.05264282226562\nEpoch 12/20, Loss: 239.94920349121094\nEpoch 13/20, Loss: 168.4589080810547\nEpoch 14/20, Loss: 295.8663024902344\nEpoch 15/20, Loss: 98.7043685913086\nEpoch 16/20, Loss: 90.49536895751953\nEpoch 17/20, Loss: 106.86287689208984\nEpoch 18/20, Loss: 79.06523132324219\nEpoch 19/20, Loss: 129.24783325195312\nEpoch 20/20, Loss: 89.89482116699219\nAverage Test Loss: 119.26110229492187\n"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/opt/dataiku/code-env/lib/python3.8/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.\n  warnings.warn(\"Setuptools is replacing distutils.\")\n"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Run 2023_11_21T17_48_06 done\n----------------------------------------\nStarting run 2023_11_21T17_48_14 ...\n(10, 128, 1, 0.01, 20)\nEpoch 1/20, Loss: 16236.279296875\nEpoch 2/20, Loss: 6466.27783203125\nEpoch 3/20, Loss: 2592.650634765625\nEpoch 4/20, Loss: 244.9362335205078\nEpoch 5/20, Loss: 199.1974334716797\nEpoch 6/20, Loss: 241.02749633789062\nEpoch 7/20, Loss: 117.8201675415039\nEpoch 8/20, Loss: 203.85336303710938\nEpoch 9/20, Loss: 155.43408203125\nEpoch 10/20, Loss: 90.15191650390625\nEpoch 11/20, Loss: 70.1321029663086\nEpoch 12/20, Loss: 62.49103927612305\nEpoch 13/20, Loss: 187.18812561035156\nEpoch 14/20, Loss: 47.10395050048828\nEpoch 15/20, Loss: 70.4273452758789\nEpoch 16/20, Loss: 48.119651794433594\nEpoch 17/20, Loss: 39.09994888305664\nEpoch 18/20, Loss: 48.169612884521484\nEpoch 19/20, Loss: 81.95384979248047\nEpoch 20/20, Loss: 11.833883285522461\nAverage Test Loss: 40.33952674865723\nRun 2023_11_21T17_48_14 done\n----------------------------------------\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 6. Deploying the model in the Flow\n\nYou can now deploy your model either via the [Experiment Tracking interface](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/viewing.html?_gl\u003d1*18yft7r*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..) or through our [Python API](https://doc.dataiku.com/dss/latest/mlops/mlflow-models/importing.html?_gl\u003d1*84hye6*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..). In either case, you will need an evaluation dataset that contains a `target`column and the features. This can be the `validation` set used in this notebook.\n\nIn the code below, if the `AUTO_DEPLOY` is set to `True`, the model that will be deployed is the one with the best metric, each time this notebook/recipe runs."
      ]
    },
    {
      "execution_count": 11,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "if AUTO_DEPLOY:\n    with project.setup_mlflow(mf) as mlflow: \n\n        # Get experiment\n        experiment \u003d mlflow.get_experiment_by_name(EXPERIMENT_NAME)\n        print(\"Experiment: \", experiment)\n\n        # List the runs and get the one with the best accuracy score\n        print(\"Looking for the best run within the experiment\")\n        best_run \u003d None\n\n        for index, run_info in mlflow.search_runs(experiment.experiment_id).iterrows():\n            run \u003d mlflow.get_run(run_info[\"run_id\"])\n            if best_run is None:\n                best_run \u003d mlflow.get_run(run_info[\"run_id\"])\n            elif run.data.metrics.get(DEPLOYMENT_METRIC, 0) \u003c best_run.data.metrics.get(DEPLOYMENT_METRIC, 0):\n                best_run \u003d run\n        print(f\"Run id {best_run.info.run_id} with {DEPLOYMENT_METRIC}\u003d{best_run.data.metrics.get(DEPLOYMENT_METRIC)}\")\n\n        # Deploy the model on the flow\n        run_id \u003d best_run.info.run_id\n        prediction_type \u003d DSSPredictionMLTaskSettings.PredictionTypes.REGRESSION\n\n        # Get or create the Saved Model\n        sm_id \u003d None\n        for sm in project.list_saved_models():\n            if sm[\"name\"] !\u003d SAVED_MODEL_NAME:\n                continue\n            else:\n                sm_id \u003d sm[\"id\"]\n                print(f\"Found Saved Model {sm[\u0027name\u0027]} with id {sm[\u0027id\u0027]}\")\n                break\n        if sm_id:\n            saved_model \u003d project.get_saved_model(sm_id)\n        else:\n            saved_model \u003d project.create_mlflow_pyfunc_model(SAVED_MODEL_NAME, PREDICTION_TYPE)\n            sm_id \u003d saved_model.id\n            print(f\"Saved Model not found, created new one with id {sm_id}\")\n\n        # Define model version\n        model_versions\u003d[model[\u0027id\u0027] for model in saved_model.list_versions()]\n\n        if model_versions is None:\n            version_id \u003d \"v1\"\n        else:\n            i\u003d1\n            print(f\"model_versions: {model_versions}\")\n            while \"v{}\".format(i) in model_versions:\n                i+\u003d1\n            version_id \u003d \"v{}\".format(i)\n        \n        # Retrieve the path of its model directory\n        model_path \u003d f\"{experiment.experiment_id}/{run_id}/artifacts/{type(model).__name__}-{run_id}\"\n        print(f\"Its model is located at: {model_path}\")\n        \n        # Import model by generating a Saved Model version\n        #best_run_id \u003d f\"{run_id}\"\n        mlflow_version \u003d saved_model.import_mlflow_version_from_managed_folder(\n            version_id\u003dversion_id,\n            managed_folder\u003dEXPERIMENT_FOLDER_ID,\n            path\u003dmodel_path,\n            code_env_name\u003dMLFLOW_CODE_ENV_NAME)\n        \n        # Make this Saved Model version the active one\n        saved_model.set_active_version(mlflow_version.version_id)\n                      \n        # Evaluate the imported model\n        mlflow_version.set_core_metadata(target_column_name\u003dTARGET_NAME,\n                                         get_features_from_dataset\u003dEVAL_DATASET)\n        mlflow_version.evaluate(EVAL_DATASET)        "
      ],
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Experiment:  \u003cExperiment: artifact_location\u003d\u0027dss-managed-folder://pshnkijv/pytorch_regression_\u0027, creation_time\u003d1699974667574, experiment_id\u003d\u0027pytorch_regression_\u0027, last_update_time\u003d1699974667574, lifecycle_stage\u003d\u0027active\u0027, name\u003d\u0027PyTorch-Regression-Model\u0027, tags\u003d{}\u003e\nLooking for the best run within the experiment\nRun id 2023_11_20T10_10_00 with test_loss\u003d33.5103572845459\nFound Saved Model PyTorch Regression Model with id ajPPDEF1\nmodel_versions: [\u0027v1\u0027, \u0027v2\u0027, \u0027v3\u0027, \u0027v4\u0027]\nIts model is located at: pytorch_regression_/2023_11_20T10_10_00/artifacts/SimpleNN-2023_11_20T10_10_00\n"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/opt/dataiku/code-env/lib/python3.8/site-packages/pandas/core/indexes/base.py:395: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\nDeprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n  elif issubclass(data.dtype.type, np.bool) or is_bool_dtype(data):\n/opt/dataiku/code-env/lib/python3.8/site-packages/pandas/core/dtypes/cast.py:1371: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. \nDeprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n  return np.object\n"
        }
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        ""
      ],
      "outputs": []
    }
  ]
}