# Amazon S3[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#amazon-s3 "Permalink to this headline")

DSS can interact with Amazon Web Services’ Simple Storage Service (AWS S3) to:

* Read and write datasets

* Read and write managed folders

S3 is an object storage service: you create “buckets” that can store arbitrary binary content and textual metadata under a specific key, unique in the container.

While not technically a hierarchical file system with folders, sub-folders and files, that behavior can be emulated by using keys containing `/`. For instance, you can store your daily logs using keys like `2015/01/24/app.log`. S3 lets you list all objects with a specific prefix, say `2015/01/` and many S3 clients (including the AWS Web Console) can “browse” your buckets this way.

DSS uses the same filesystem-like mechanism when accessing S3: when you specify a bucket, you can browse it to quickly find your dataset, or you can set the prefix in which DSS may output datasets. Datasets on S3 thus must be in one of the supported filesystem formats.

Note

* Using S3 as a filesystem comes with a few limitations:: * keys must not start with a `/`
* “files” with names containing `/` are not supported
* “folders” (prefixes) `.` and `..` are not supported
* like on a filesystem, a file and a folder with the same name are not
supported: if a file `some/key` exists, it takes precedence over a
`some/key/` prefix / folder

## Create a S3 connection[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#create-a-s3-connection "Permalink to this headline")

To create a S3 connection, you need to have a AWS keypair (access key and secret key). Note that if your DSS server is running on AWS itself, DSS can automatically retrieve credentials from the environment. In that case, you can use “Environment” credentials

### Required S3 permissions[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#required-s3-permissions "Permalink to this headline")

The access you specify in an AWS connection must have the following permissions:

* To read data from a bucket, it must at least have listing and reading

permissions on that bucket:

§ s3:ListBucket arn:aws:s3:::examplebucket

§ s3:GetObject arn:aws:s3:::examplebucket/*

* To write data to a bucket, it must also have writing, deletion and

multipart-upload-aborting permissions on that bucket:

§ s3:ListBucket arn:aws:s3:::examplebucket

§ s3:GetObject arn:aws:s3:::examplebucket/*

§ s3:PutObject arn:aws:s3:::examplebucket/*

§ s3:DeleteObject arn:aws:s3:::examplebucket/*

§ s3:AbortMultipartUpload arn:aws:s3:::examplebucket/*

* Not mandatory, but bucket listing permission allows you to pick a bucket

from a dropdown list instead of manually typing its name when creating a

dataset:

§ s3:ListAllMyBuckets arn:aws:s3:::*

* Also not mandatory, bucket locating permission can improve performance by

making DSS access a bucket via its preferred AWS endpoint:

§ s3:GetBucketLocation arn:aws:s3:::*

For more details about how to create policies and add these permissions to them, please see the AWS documentation here: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/set-permissions.html

### Transfer ownership to the bucket owner[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#transfer-ownership-to-the-bucket-owner "Permalink to this headline")

If the target bucket is owned by another account, and if ACLs are enabled and set to “Bucket owner preferred”, it is possible to transfer the ownership of uploaded data to the bucket owner by adding the `bucket-owner-full-control` canned ACL to the write operation.

The `bucket-owner-full-control` canned ACL can be set on every write operation in DSS by setting the `dku.connection.s3.addBucketOwnerCannedACL` custom property to `true` on the S3 connection.

In addition to the required permissions mentioned above to write data to a bucket, the AWS account used by the S3 connection must have permission to Read/Write ACLs in the target bucket. This can be set on the bucket by its owner.

For more details about configuring ACLs, please refer to the AWS documentation here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/managing-acls.html

Note

The recommendation for cross-account access is to use AssumeRole. For more details about AssumeRole, please see the AWS documentation here: https://docs.aws.amazon.com/STS/latest/APIReference/API\_AssumeRole.html

## Creating S3 datasets[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#creating-s3-datasets "Permalink to this headline")

After creating your S3 connection in Administration, you can create S3 datasets.

From either the Flow or the datasets list, click on New dataset > S3.

* Select the connection in which your files are located

* If available, select the bucket (either by listing or entering it)

* Click on “Browse” to locate your files.

## Connections path handling[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#connections-path-handling "Permalink to this headline")

The S3 connection can be either in “free selection” mode, or in “path restriction mode”.

In “free selection” mode, users can select the bucket in which they want to read, and the path within the bucket. If the AWS keypair has the permission to list buckets, a bucket selector will be available for users.

In “path restriction mode”, you choose a bucket, and optionally a path within the bucket. Users will only be able to read and write data within that “base bucket + path”.

To enable “path restriction mode”, simply write a bucket name (and optionally a path in bucket) in the “Path restrictions” section of the connection settings

## Location of managed datasets and folders[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#location-of-managed-datasets-and-folders "Permalink to this headline")

### For a “free selection” connection[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#for-a-free-selection-connection "Permalink to this headline")

When you create a managed dataset or folder in a S3 connection, DSS will automatically create it within the “Default bucket” and the “Default path”.

Below that root path, the “naming rule” applies. See Making relocatable managed datasets for more information.

### For a “path restriction” connection[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#for-a-path-restriction-connection "Permalink to this headline")

When you create a managed dataset or folder in a S3 connection, DSS will automatically create it within the Bucket and Path selected in the “Path restrictions” section, and will append the “Default path” from the “managed datasets & folders” section.

Below that root path, the “naming rule” applies. See Making relocatable managed datasets for more information.

## Server-side encryption of files[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#server-side-encryption-of-files "Permalink to this headline")

### Encryption Mode[¶](https://doc.dataiku.com/dss/latest/connecting/s3.html#encryption-mode "Permalink to this headline")

Options for server-side encryption of files can be set to either ‘None’, ‘SSE-S3’ or ‘SSE-KMS’. For more information on these options, please see the AWS documentation here: https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
