# Dynamic Google Dataproc clusters[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#dynamic-google-dataproc-clusters "Permalink to this headline")

* Prerequisites and limitations

* Create your first cluster

+ Before Running

+ Install the plugin

+ Define GCS connections

+ Create the cluster and configure it

+ Use your cluster

+ Stop the cluster

* Using dynamic Dataproc clusters for scenarios

* Cluster actions

+ Manual run

+ As part of a scenario

* Advanced settings

+ Hive metastore

+ Labels

Warning

**Tier 2 support**: Management of dynamic Dataproc clusters is provided through a plugin and is covered by Tier 2 support

DSS does not support the latest Dataproc versions, security options are limited, and our experience is that Dataproc deployments are associated swith a higher complexity and tend to generate higher administration workloads.

We recommend that you use a fully Elastic AI infrastructure based on GKE. Please see Elastic AI computation, or get in touch with your Dataiku Customer Success Manager, Technical Account Manager or Sales Engineer for more information and studying the best options.

DSS can create and manage multiple Dataproc clusters, allowing you to easily scale your workloads across multiple clusters, dynamically create and scale clusters for some scenarios, etc.

For more information on dynamic clusters and the usage of a dynamic cluster for a scenario, please see Multiple Hadoop clusters.

Support for dynamic clusters is provided through the “Dataproc clusters” DSS plugin. You will need to install this plugin in order to use this feature.

## Prerequisites and limitations[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#prerequisites-and-limitations "Permalink to this headline")

* Like for other kinds of multi-cluster setups, the server that runs DSS needs to have the client libraries for the proper Hadoop distribution. In that case, your server needs to have the Dataproc client libraries for the version of Dataproc you will use.

* When working with multiple clusters, all clusters should run the same Dataproc version. If running different versions, some incompatibilities may occur.

## Create your first cluster[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#create-your-first-cluster "Permalink to this headline")

### Before Running[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#before-running "Permalink to this headline")

The Google Compute Engine (GCE) VM that hosts DSS is associated with a given service account, which MUST have the following AMI permissions to run properly:

* Dataproc Editor

* Dataproc Service Agent

Otherwise the user account running DSS must have the required credentials in order to create Dataproc clusters.

### Install the plugin[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#install-the-plugin "Permalink to this headline")

From the “Plugins” section of the Administration panel, go to the store and install the Dataproc clusters plugin.

### Define GCS connections[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#define-gcs-connections "Permalink to this headline")

Most of the time, when using dynamic Dataproc clusters, you will store all inputs and outputs of your flows on GCS. Access to GCS from DSS and from the Dataproc cluster is done through GCS.

* Go to Administration > Connections, and add a new HDFS connection

* Enter “gs://your-bucket” or “gs://your-bucket/prefix” as the root path URI

Unless the DSS host has implicit access to the bucket through its IAM role or default credentials in environment variable `GOOGLE\_APPLICATION\_CREDENTIALS`, define connection-level credentials in “Extra Hadoop conf”:

* Add a property called “spark.hadoop.google.cloud.auth.service.account.enable” with value true

* Add a property called “spark.hadoop.google.cloud.auth.service.account.json.keyfile” with path to your keyfile on the server

### Create the cluster and configure it[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#create-the-cluster-and-configure-it "Permalink to this headline")

Go to Administration > Cluster and click “Create cluster”

In “Type”, select “Dataproc cluster (create cluster)” and give a name to your new cluster. You are taken to the “managed cluster” configuration page, where you will set all of your Dataproc cluster settings.

You will then need to fill in the following fields:

* The GCP project ID in which to build the Dataproc Cluster

* The cluster name on GCP

* The DataProc version to use (MUST be the same Linux distribution and version as your GCE VM)

* The desired region/zone

* The network tags you want to attach to your cluster nodes

* The instance type for the master and worker nodes

* The total number of worker nodes you require

Click on “Start/Attach”. Your Dataproc cluster is created. This phase generally lasts 5 to 10 minutes. When the progress modal closes, you have a working Dataproc cluster, and an associated DSS dynamic cluster definition that can talk to it.

### Use your cluster[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#use-your-cluster "Permalink to this headline")

In any project, go to Settings > Cluster, and select the identifier you gave to the Dataproc cluster. Any recipe or Hive notebook running in this project will now use your Dataproc cluster.

### Stop the cluster[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#stop-the-cluster "Permalink to this headline")

Go to Administration > Clusters > Your cluster and click “Stop/Detach” to destroy the Dataproc cluster and release resources.

Note that the DSS cluster definition itself remains, allowing you to recreate the Dataproc cluster at a later time. Projects that are configured to use this cluster while it is in “Stopped/Detached” state will fail.

## Using dynamic Dataproc clusters for scenarios[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#using-dynamic-dataproc-clusters-for-scenarios "Permalink to this headline")

For a fully elastic approach, you can create Dataproc clusters at the beginning of a sequence of scenarios, run the scenarios and then destroy the Dataproc cluster, fully automatically.

Please see Multiple Hadoop clusters for more information. In the “Setup Cluster” scenario step, you will need to enter the Dataproc cluster configuration details

## Cluster actions[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#cluster-actions "Permalink to this headline")

In addition to the basic “Start” and “Stop”, the Dataproc clusters plugin provides the ability to scale up and down an attached Dataproc cluster.

### Manual run[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#manual-run "Permalink to this headline")

Go to the “Actions” tab of your cluster, and select the “Scale cluster up or down” macro. You will then choose the new number of desired worker nodes, with the option of creating spot (pre-emptible) nodes.

### As part of a scenario[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#as-part-of-a-scenario "Permalink to this headline")

You can scale up/down a cluster as part of a scenario

* Add an “Execute macro” step

* Select the “Scale cluster up or down” macro

* Enter the DSS cluster identifier, either directly, or using a variable - the latter case is required if you setup your cluster as part of the scenario.

* Select the settings

In this kind of setup you will generally scale up at the beginning of the scenario and scale down at the end. You will need two steps for that. Make sure to select “Run this step > Always” for the “scale down” step. This way, even if your scenario fails, the scale down operation will be executed.

## Advanced settings[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#advanced-settings "Permalink to this headline")

In addition to the basic settings outlined above, the Dataproc clusters plugin provides some advanced settings:

### Hive metastore[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#hive-metastore "Permalink to this headline")

* Metastore database type (transient or persisted)

* Databases to use

### Labels[¶](https://doc.dataiku.com/dss/latest/hadoop/dynamic-dataproc.html#labels "Permalink to this headline")

You can add labels to your cluster. Variables expansion is not supported here.
