Amazon S3
The Amazon S3 integration shows how to set up a connection to the Amazon S3 object storage using Python. We demonstrate how you can download files from the storage and vice versa. Before your Python script can interact with the Amazon S3 storage, we first need to get the credentials.
Get access keys for Amazon
To connect to Amazon S3 object storage, you need an access key id
and secret access key
.
AWS CLI
If you have the AWS CLI, you can use it to interactive create a configuration file. In your terminal run:
aws configure
Follow the prompts. In the end, you can find your credentials in the file ~/.aws/credentials/
.
AWS Console
For instructions about creating a user using the IAM Console, follow the steps on Creating IAM users.
With an IAM user, you can get access keys by following the steps in Managing access keys.
Python
In the example below, we show how to connect the Amazon S3 object storage using Python. We demonstrate how you can download files from the storage and vice versa. To do this, we need Python 3.6 or newer (See how to install Python). In a virtual environment, you should install:
- boto3 (required to set up the connection)
- python-dotenv
We recommend using a requirements.txt file. In this file, you can add:
boto3 # tested with version 1.20.4
python-dotenv # tested with version 0.18.0
Add AskAnna project variables
To interact with Amazon Web Services, you first need to set the access keys as variables. In the AskAnna project, you should add project variables. On the project page, go to the tab variables. Here you can create new variables. For how to get the values of these variables, see Get access keys for Amazon.
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
Warning
Make sure to set the variables to masked. You don't want to expose these values.
Configure dotenv
In the Python examples, we use:
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())
These two lines make it a lot easier to develop your Python code locally, while you can also run the same code in AskAnna. When you add project variables, these variables will become available as environment variables in the run environment. Locally, you can add a file .env
. When you run the Python code locally, the environment variables are loaded from this file. Read more about this on the project page python-dotenv.
To run the above example, you need a .env
file with:
AWS_ACCESS_KEY_ID={ACCESS_KEY}
AWS_SECRET_ACCESS_KEY={SECRET_KEY}
For how to get the values of these variables, see Get access keys for Amazon.
Download from Amazon S3
See also Downloading files in the Boto3 documentation.
import os
import boto3
# More about dotenv in the section `Configure dotenv`
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())
bucket_name = "S3 BUCKET" # Bucket to upload to`
object_name = "OBJECT NAME" # S3 OBJECT NAME
file_name = "PATH TO FILE" # File to save to
s3_client = boto3.client(
"s3",
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
)
s3_client.download_file(bucket_name, object_name, file_name)
Uploading to Amazon S3
See also Uploading files in the Boto3 documentation.
import os
import boto3
from botocore.exceptions import ClientError
# More about dotenv in the section `Configure dotenv`
from dotenv import find_dotenv, load_dotenv
load_dotenv(find_dotenv())
file_name = "PATH TO FILE" # File to upload
bucket_name = "S3 BUCKET" # Bucket to upload to`
s3_client = boto3.client(
"s3",
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
)
try:
response = s3_client.upload_file(file_name, bucket_name)
except ClientError as e:
print(e)
More examples
In the Boto3 documentation, you can find more Amazon S3 examples.