Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction

S3 (Simple Storage Service) is a highly scalable, object-based cloud storage service used for storing and retrieving data.

There are different way of performing actions on files (e.g. upload, download, read, delete) or bucket specifically (e.g. create bucket) on s3 buckets:

With S3 buckets, accessing data is even easier than before. For those who want to use Python (encouraged), it is easy as pie.

Make sure you have these ready:

  1. Python 3
  2. Project ID of the bucket you want to mount
  3. Access key and secret access key for the bucket

Take a look at this code segment, which allows you to access the bucket, list its objects and upload/download files from it. Feel free to copy paste this segment and fill in with your own values.

Code Block
languagepy
import os
import io
import boto3
from botocore.session import Session
from botocore.handlers import validate_bucket_name

#Initializing some values 
project_id = '123' #Fill this in 
bucketname = 'MyFancyBucket123'  #Fill this in  
bucketname = project_id + ':' + bucketname
access_key = '123asdf'  #Fill this in  
secret_access_key = '123asdf111'  #Fill this in  
#This is a neat trick to allow us to specify our bucket name in terms of ProjectID:bucketname
botocore_session = Session()
botocore_session.unregister('before-parameter-build.s3', validate_bucket_name)
boto3.setup_default_session(botocore_session = botocore_session)

#Initialize the S3 client
s3 = boto3.client('s3', endpoint_url='https://s3.waw3-1.cloudferro.com',
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_access_key)

#List all the objects in the bucket
response = s3.list_objects(Bucket=bucketname)
for item in response['Contents']:
    print(item['Key'])

#Read a file into Python's memory and open it as a string
filename = '/folder1/folder2/myfile.txt'  #Fill this in  
obj = s3.get_object(Bucket=bucketname, Key=filename)
myObject = obj['Body'].read().decode('utf-8') 
print(myObject)

#Downloading a file from the bucket
with open('myfile', 'wb') as f:  #Fill this in  
    s3.download_fileobj(bucketname, 'myfile.txt', f) 

#Uploading a file to the bucket (make sure you have write access)
response = s3.upload_file('myfile', bucketname, 'myfile')  #Fill this in  

result = s3.list_objects(Bucket = bucket, Prefix='/')
for o in result.get('Contents'):
    data = s3.get_object(Bucket=bucket, Key=o.get('Key'))
    contents = data['Body'].read()
    print(contents.decode("utf-8")) 

Check out a full code example at the official boto3 website: 

...



Related articles

Content by Label
showLabelsfalse
max5
spacesEWCLOUDKB
showSpacefalse
sortmodified
reversetrue
typepage
cqllabel in ("python","s3","kb-how-to-article","how-to","howto") and type = "page" and space = "EWCLOUDKB"
labelspython s3

...