With S3 buckets, accessing data is even easier than before. For those who want to use Python (encouraged), it is easy as pie.
If you're connecting to EUMETSAT buckets, take note of (1).
Make sure you have these ready:
...
Take a look at this code segment, which allows you to access the bucket, list its objects and upload/download files from it.
Start by declaring some initial values for boto3 to know where your bucket is located at. Feel free to copy paste this segment and fill in with your own values.
Code Block | ||
---|---|---|
| ||
import os import io import boto3 from botocore.session import Session from botocore.handlers import validate_bucket_name #Initializing some values project_id = '123' #Fill this in bucketname = 'MyFancyBucket123' #Fill this in bucketname = project_id + ':' + bucketname access_key = '123asdf' #Fill this in secret_access_key = '123asdf111' #Fill this in endpoint = 'https://my-s3-endpoint.com' #Fill this in |
(1) If you aren't connecting to EUMETSAT buckets, or a bucket someone shared with you, you can probably skip this part. Now, if the bucket is located at another project (e.g. someone is sharing their bucket with you), you probably need to denote the bucket name with project_id:bucketname. For boto3 to be able to use that kind of naming convention, we need to use a small trick.
Code Block |
---|
#This is a neat trick that allows us to specify our bucket name in terms of ProjectID:bucketname bucketname = project_id + ':' + bucketname botocore_session = Session() botocore_session.unregister('before-parameter-build.s3', validate_bucket_name) boto3.setup_default_session(botocore_session = botocore_session) |
Now lets start by initializing the S3 client with our access keys and endpoint:
Code Block |
---|
#Initialize the S3 client s3 = boto3.client('s3', endpoint_url=endpoint, aws_access_key_id = access_key, aws_secret_access_key = secret_access_key) |
As a first step, and to confirm we have successfully connected, lets list the objects inside our bucket (up to a 1.000 objects).
Code Block |
---|
#List) #List all the objects in theour bucket response = s3.list_objects(Bucket=bucketname) for item in response['Contents']: print(item['Key']) |
Now lets try to read a file from a bucket into Python's memory, so we can work with it inside Python without ever saving the file on our local computer:
Code Block |
---|
#Read a file into Python's memory and open it as a string filename = '/folder1/folder2/myfile.txt' #Fill this in obj = s3.get_object(Bucket=bucketname, Key=filename) myObject = obj['Body'].read().decode('utf-8') print(myObject) |
But if you'd want to download the file instead of reading it into memory, here's how you'd do that:
Code Block |
---|
#Downloading a file from the bucket
with open('myfile', 'wb') as f: #Fill this in
s3.download_fileobj(bucketname, 'myfile.txt', f) |
And similarly you can upload files to the bucket (given that you have write access to the bucket):
Code Block |
---|
#Uploading a file to the bucket (make sure you have write access) response = s3.upload_file('myfile', bucketname, 'myfile') #Fill this in result = s3.list_objects(Bucket = bucket, Prefix='/') for o in result.get('Contents'): data = s3.get_object(Bucket=bucket, Key=o.get('Key')) contents = data['Body'].read() print(contents.decode("utf-8")) |
If you're interested in more, I recommend taking a look at this article, which gives you a more detailed view into boto3's functionality:
https://dashbird.io/blog/boto3-aws-python/
Check out a full code example at the official boto3 website:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-examples.html
You can also see a differently styled tutorial at:
https://towardsdatascience.com/introduction-to-pythons-boto3-c5ac2a86bb63
Related articles
Content by Label | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...