{
  "metadata": {
    "kernelspec": {
      "name": "py-dku-venv-py_39_mlflow_in_dku",
      "display_name": "Python (env py_39_mlflow_in_dku)",
      "language": "python"
    },
    "hide_input": false,
    "language_info": {
      "name": "python",
      "version": "3.9.20",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "createdOn": 1652730934978,
    "tags": [
      "recipe-editor"
    ],
    "customFields": {},
    "creationTag": {
      "versionNumber": 0,
      "lastModifiedBy": {
        "login": "harizo"
      },
      "lastModifiedOn": 1652730934978,
      "extendedProperties": {}
    },
    "dkuGit": {
      "lastInteraction": 0
    },
    "associatedRecipe": "recipe_from_notebook_xptrackingsklearn",
    "creator": "harizo",
    "modifiedBy": "camille.cochener"
  },
  "nbformat": 4,
  "nbformat_minor": 1,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Experiment Tracking for tabular classification with Scikit-learn\n\nIn this notebook, you can:\n\n1. Train scikit-learn classification models\n2. Log those models in the MLflow format and experiments details so that they can be compared using the DSS [Experiment Tracking interface](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/viewing.html?_gl\u003d1*shnz4d*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Preparing the experiment\n\nThe first step is to prepare the ground for the experiment tracking:\n\nThe following code imports all libraries and defines constant variables, handles and function necessary to the training and tracking of scikit-learn models. \n\nMake sure to change the ids and names to your own parameters.\n\nFor more information regarding experiment tracking in code, refer to our [documentation](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/tracking.html?_gl\u003d1*124ki6d*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..)."
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# -*- coding: utf-8 -*-\n\n# Import necessary libraries\nimport dataiku\nimport pandas as pd\nimport mlflow\nimport warnings\nfrom datetime import datetime\nfrom sklearn.preprocessing import OrdinalEncoder\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.model_selection import cross_validate, StratifiedKFold, ParameterGrid\nfrom dataikuapi.dss.ml import DSSPredictionMLTaskSettings\n\n# Constants for experiment tracking and model management\nEXPERIMENT_TRACKING_FOLDER_NAME \u003d \"Tabular Classification Experiments\"\nEXPERIMENT_TRACKING_FOLDER_CONNECTION \u003d \"filesystem_folders\"\nEXPERIMENT_NAME \u003d \"Tabular-Experiments\"\nMLFLOW_CODE_ENV_NAME \u003d \"py_39_mlflow_in_dku\"\nSAVED_MODEL_NAME \u003d \"Tabular Classification Model\"\nDATASET_TRAINING \u003d \"input-tabular_classification\"\nDATASET_EVALUATION \u003d \"evaluate-tabular_classification\"\nAUTO_DEPLOY\u003dTrue\n\n# Utility function to get current timestamp as a string\ndef now_str() -\u003e str:\n    return datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ],
      "outputs": []
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Get a handle on the project\nclient \u003d dataiku.api_client()\nproject \u003d client.get_default_project()\n\n# Initialize MLflow extension and client\nmlflow_extension \u003d project.get_mlflow_extension()"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Loading the data\n\nThe second step to training classification models is to obtain the data.\n\nHere, the datasets are loaded from the Flow using the DSS Python API:\n- A `train` dataset will be used to train the classifiers\n- A `validation` dataset will be used to assess the performance of the classifiers"
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Load your data into a Pandas DataFrame\ntraining_dataset \u003d dataiku.Dataset(DATASET_TRAINING)\ndf \u003d training_dataset.get_dataframe()\n\n# Define target variable and features\ntarget_name \u003d \"y\"\ntarget \u003d df[target_name]\ndata \u003d df.drop(columns\u003d[target_name])\n\n# Define categorical columns\ncategorical_cols \u003d [\"job\", \"marital\", \"education\", \"default\", \"housing\", \"loan\", \"contact\", \"month\", \"poutcome\"]"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Experiment Tracking Folder Setup\n\nNow check for the existence of the experiment tracking folder or create one"
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Check for the existence of the experiment tracking folder or create one\nproject_folders \u003d project.list_managed_folders()\nfolder \u003d None\nif len(project_folders) \u003e 0:\n    for mf in project_folders:\n        if mf[\"name\"] \u003d\u003d EXPERIMENT_TRACKING_FOLDER_NAME:\n            folder_id \u003d mf[\"id\"]\n            print(f\"Found experiment tracking folder {EXPERIMENT_TRACKING_FOLDER_NAME} with id {mf[\u0027id\u0027]}\")\n            folder \u003d project.get_managed_folder(odb_id\u003dfolder_id)\n            break\n        else:\n            continue\n    # If you reach this point, you didn\u0027t find the experiment tracking folder among the existing ones.\n    if not folder:\n        print(\"Experiment tracking folder not found. Creating it...\")\nelse:\n    print(\"No folder found in project. Creating one for experiment tracking...\")\n    # Write the creation of the mf code here.\n    folder \u003d project.create_managed_folder(EXPERIMENT_TRACKING_FOLDER_NAME,\n                                       connection_name\u003dEXPERIMENT_TRACKING_FOLDER_CONNECTION)"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Model and hyperparameter grid\n\nNow define the hyperparameter grid to go through, the type of validation and the performance metrics to compute."
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define hyperparameters for the RandomForestClassifier\nhparams_dict \u003d {\n    \"n_estimators\": [50, 100],\n    \"criterion\": [\"gini\"],\n    \"max_depth\": [12, 20],\n    \"min_samples_split\": [3],\n    \"random_state\": [42]\n}\n\n# Turn the hyperparameters dict to a grid\nparam_grid \u003d ParameterGrid(hparams_dict)\n\n# Define the number of cross-validation folds\nn_cv_folds \u003d 5\ncv \u003d StratifiedKFold(n_splits\u003dn_cv_folds)\n\n# Define the metrics to evaluate during cross-validation\nmetrics \u003d [\"f1_macro\", \"roc_auc\"]"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Experiment runs\n\nAt last, the following piece of code runs training while tracking all the experiments. After a successful run, you will be able to deploy the model either visually or programmatically.\n\nThe following code can be split into the different steps:\n\n1. Create an `mlflow` context using the `setup_mlflow()` method from the Dataiku API.\n2. Create an experiment\n3. Loop through the list of hyperparameter combinations so that for each combination, you start a run in which you:\n - Create and train a model.\n - Collect trained model metrics on test set\n - Serialize + save the model. This is a necessary intermediary step.\n - Log the model along with the collected metrics and other model metadata (hyperparemeters, epochs, code environment…)."
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Start an MLflow run and log parameters, metrics, and model\nrun_ts \u003d now_str()\nrun_name \u003d f\"run-{run_ts}\"\n\nwith project.setup_mlflow(managed_folder\u003dfolder) as mlflow:\n\n    # If the experiment did not already exist, we create it\n    if len(mlflow_extension.list_experiments()) \u003e\u003d 1:\n        if EXPERIMENT_NAME in [exp[\u0027name\u0027] for exp in mlflow_extension.list_experiments()[\u0027experiments\u0027]]:\n            print(\"Experiment already existed with name : {}\".format(EXPERIMENT_NAME))\n        else:\n            experiment_id \u003d mlflow.create_experiment(EXPERIMENT_NAME)\n            print(\"Experiment created with name : {}\".format(EXPERIMENT_NAME))\n    else:\n        experiment_id \u003d mlflow.create_experiment(EXPERIMENT_NAME)\n        print(\"Experiment created with name : {}\".format(EXPERIMENT_NAME))\n\n    # For display\n    experiment \u003d mlflow.get_experiment_by_name(EXPERIMENT_NAME)\n    experiment_id \u003d experiment.experiment_id\n    print(\"Experiment name is: {} and ID is: {} \".format(EXPERIMENT_NAME, experiment_id))\n\n    mlflow.set_experiment(EXPERIMENT_NAME)\n                  \n    for hparams in param_grid:\n        with mlflow.start_run(run_name\u003drun_name) as run:\n            run_id \u003d run.info.run_id\n            # Create dictionaries to store run parameters and metrics\n            run_params \u003d {}\n            run_metrics \u003d {}\n            print(f\"Starting run {run_name} (id: {run_id})...\")\n\n            # Define preprocessing for categorical columns\n            run_params[\"categorical_cols\"] \u003d categorical_cols\n            categorical_preprocessor \u003d OrdinalEncoder(handle_unknown\u003d\"use_encoded_value\", unknown_value\u003d-1)\n            preprocessor \u003d ColumnTransformer([(\u0027categorical\u0027, categorical_preprocessor, categorical_cols)], remainder\u003d\"passthrough\")\n            \n            # Initialize the classifier with the specified hyperparameters\n            clf \u003d RandomForestClassifier(**hparams)\n            \n            # Create a pipeline that includes preprocessing and the classifier\n            pipeline \u003d make_pipeline(preprocessor, clf)\n\n            # Store model algorithm name and hyperparameters in run_params\n            model_algo \u003d type(clf).__name__\n            run_params[\"model_algo\"] \u003d model_algo\n            for hp in hparams.keys():\n                run_params[hp] \u003d hparams[hp]\n\n            # Print the parameters and metrics to log\n            print(f\"Parameters to log:\\n {run_params}\")\n            print(100 * \u0027-\u0027)\n            print(f\"Metrics to log:\\n {metrics}\")\n\n            # Perform cross-validation\n            print(f\"Running cross-validation...\")\n            scores \u003d cross_validate(pipeline, data, target, cv\u003dcv, scoring\u003dmetrics)\n            for m in [f\"test_{mname}\" for mname in metrics]:\n                run_metrics[f\"mean_{m}\"] \u003d scores[m].mean()\n                run_metrics[f\"std_{m}\"] \u003d scores[m].std()\n\n            # Fit the pipeline on the entire training data\n            pipeline.fit(X\u003ddata, y\u003dtarget)\n\n            # Log the class labels order\n            run_params[\"class_labels\"] \u003d pipeline.classes_.tolist()\n\n            # Log parameters, metrics, and the trained model to MLflow\n            mlflow.log_params(params\u003drun_params)\n            mlflow.log_metrics(metrics\u003drun_metrics)\n            artifact_path \u003d f\"{model_algo}-{run_id}\"\n            mlflow.sklearn.log_model(sk_model\u003dpipeline, artifact_path\u003dartifact_path)\n\n            # Set additional information to facilitate run promotion\n            mlflow_extension.set_run_inference_info(\n                run_id\u003drun_id,\n                prediction_type\u003d\"BINARY_CLASSIFICATION\",\n                classes\u003drun_params[\"class_labels\"],\n                code_env_name\u003dEXPERIMENT_NAME,\n                target\u003d\"y\"\n            )\n\n            # Print completion message with artifact URI\n            print(f\"DONE! Your artifacts are available at {run.info.artifact_uri}\")"
      ],
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 6. Deploying the model in the Flow\n\nYou can now deploy your model either via the [Experiment Tracking interface](https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/viewing.html?_gl\u003d1*18yft7r*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..) or through our [Python API](https://doc.dataiku.com/dss/latest/mlops/mlflow-models/importing.html?_gl\u003d1*84hye6*_ga*MTQwMDcxMzEwNy4xNjgzMjg4OTI5*_ga_B3YXRYMY48*MTY5OTk3MjA1OC4yOTkuMC4xNjk5OTcyMDYwLjU4LjAuMA..). In either case, you will need an evaluation dataset that contains a `target`column and the features. This can be the `validation` set used in this notebook.\n\nIn the code below, if the `AUTO_DEPLOY` is set to `True`, the model that will be deployed is the one with the best metric, each time this notebook/recipe runs."
      ]
    },
    {
      "execution_count": 0,
      "cell_type": "code",
      "metadata": {},
      "source": [
        "DEPLOYMENT_METRIC \u003d \"mean_test_roc_auc\"\n\nif AUTO_DEPLOY:\n    with project.setup_mlflow(managed_folder\u003dfolder) as mlflow:\n            best_run \u003d None\n\n            for index, run_info in mlflow.search_runs(experiment.experiment_id).iterrows():\n                run \u003d mlflow.get_run(run_info[\"run_id\"])\n                if best_run is None:\n                    best_run \u003d mlflow.get_run(run_info[\"run_id\"])\n                elif run.data.metrics.get(DEPLOYMENT_METRIC, 0) \u003c best_run.data.metrics.get(DEPLOYMENT_METRIC, 0):\n                    best_run \u003d run\n            best_run_id \u003d best_run.info.run_id\n            print(f\"Best Run id {best_run_id} with {DEPLOYMENT_METRIC}\u003d{best_run.data.metrics.get(DEPLOYMENT_METRIC)}\")\n\n            # Retrieve the path of the model directory for the best run\n            best_run_info \u003d mlflow.get_run(best_run_id).info\n            model_path \u003d f\"{experiment.experiment_id}/{best_run_id}/artifacts/{model_algo}-{best_run_id}\"\n            print(f\"Its model is located at: {model_path}\")\n\n    # Check for an existing Saved Model or create a new one if it doesn\u0027t exist\n    sm_id \u003d None\n    for sm in project.list_saved_models():\n        if sm[\"name\"] !\u003d SAVED_MODEL_NAME:\n            continue\n        else:\n            sm_id \u003d sm[\"id\"]\n            print(f\"Found Saved Model {sm[\u0027name\u0027]} with id {sm[\u0027id\u0027]}\")\n            break\n\n    # Create a new Saved Model if it was not found\n    if sm_id:\n        sm \u003d project.get_saved_model(sm_id)\n    else:\n        sm \u003d project.create_mlflow_pyfunc_model(\n            name\u003dSAVED_MODEL_NAME,\n            prediction_type\u003dDSSPredictionMLTaskSettings.PredictionTypes.BINARY\n        )\n        sm_id \u003d sm.id\n        print(f\"Saved Model not found, created new one with id {sm_id}\")\n\n    # Import the best model version from MLflow into the Saved Model\n    version_id \u003d f\"{best_run_id}\"\n    mlflow_version \u003d sm.import_mlflow_version_from_managed_folder(\n        version_id\u003dversion_id,\n        managed_folder\u003dfolder,\n        path\u003dmodel_path,\n        code_env_name\u003dMLFLOW_CODE_ENV_NAME\n    )\n\n    # Set the imported model version as the active version\n    sm.set_active_version(mlflow_version.version_id)\n\n    # Evaluate the imported model using the evaluation dataset\n    mlflow_version.set_core_metadata(\n        target_column_name\u003dtarget_name,\n        class_labels\u003d[\"no\", \"yes\"],  # Update this with dynamic class labels if needed\n        get_features_from_dataset\u003dDATASET_EVALUATION\n    )\n    mlflow_version.evaluate(DATASET_EVALUATION)"
      ],
      "outputs": []
    }
  ]
}