Amazon S3

The Amazon S3 integration shows how to set up a connection to the Amazon S3 object storage using Python. We demonstrate how you can download files from the storage and vice versa. Before your Python script can interact with the Amazon S3 storage, we first need to get the credentials.

Get access keys for Amazon

To connect to Amazon S3 object storage, you need an access key id and secret access key.

AWS CLI

If you have the AWS CLI, you can use it to interactive create a configuration file. In your terminal run:

aws configure

Follow the prompts. In the end, you can find your credentials in the file ~/.aws/credentials/.

AWS Console

For instructions about creating a user using the IAM Console, follow the steps on Creating IAM users.

With an IAM user, you can get access keys by following the steps in Managing access keys.

Python

In the example below, we show how to connect the Amazon S3 object storage using Python. We demonstrate how you can download files from the storage and vice versa. To do this, we need Python 3.6 or newer (See how to install Python). In a virtual environment, you should install:

• boto3 (required to set up the connection)
• python-dotenv

We recommend using a requirements.txt file. In this file, you can add:

boto3  # tested with version 1.20.4
python-dotenv  # tested with version 0.18.0

Add AskAnna project variables

To interact with Amazon Web Services, you first need to set the access keys as variables. In the AskAnna project, you should add project variables. On the project page, go to the tab variables. Here you can create new variables. For how to get the values of these variables, see Get access keys for Amazon.

1. AWS_ACCESS_KEY_ID
2. AWS_SECRET_ACCESS_KEY

Configure dotenv

In the Python examples, we use:

from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())

These two lines make it a lot easier to develop your Python code locally, while you can also run the same code in AskAnna. When you add project variables, these variables will become available as environment variables in the run environment. Locally, you can add a file .env. When you run the Python code locally, the environment variables are loaded from this file. Read more about this on the project page python-dotenv.

To run the above example, you need a .env file with:

AWS_ACCESS_KEY_ID={ACCESS_KEY}
AWS_SECRET_ACCESS_KEY={SECRET_KEY}

For how to get the values of these variables, see Get access keys for Amazon.

Download from Amazon S3

See also Downloading files in the Boto3 documentation.

import os
import boto3

# More about dotenv in the section `Configure dotenv`
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())


bucket_name = "S3 BUCKET"  # Bucket to upload to`
object_name = "OBJECT NAME"  # S3 OBJECT NAME
file_name = "PATH TO FILE"  # File to save to

s3_client = boto3.client(
    "s3",
    aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
)

s3_client.download_file(bucket_name, object_name, file_name)

Uploading to Amazon S3

See also Uploading files in the Boto3 documentation.

import os
import boto3
from botocore.exceptions import ClientError

# More about dotenv in the section `Configure dotenv`
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())


file_name = "PATH TO FILE"  # File to upload
bucket_name = "S3 BUCKET"  # Bucket to upload to`

s3_client = boto3.client(
    "s3",
    aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
    aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
)

try:
    response = s3_client.upload_file(file_name, bucket_name)
except ClientError as e:
    print(e)

More examples

In the Boto3 documentation, you can find more Amazon S3 examples.