...
You may also use this guide to use any other S3 storage service such as AWS from your instances at the European Weather Cloud. Just adapt the host and credential information accordingly.
Table of Contents |
---|
Managing your Object Storage with S3cmd
S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol.
Many other advanced tools (e.g. https://rclone.org/) exist, as do APIs for many languages, but this article aims only to demonstrate the basics.
Install the tool
The easiest is to install it through the system package manager
...
Alternatively, you may get the latest version from PyPi
Configure s3cmd
You will need to configure s3cmd
before you can use it. The tool will read the configuration on ~/.s3cfg
Create the configuration file if it does not exist:
No Format touch ~/.s3cfg
Edit the file and set up at least the following parameters.
ECMWF:
- ECMWF CCI1 endpoint:
host_base = object-store.os-api.cci1.ecmwf.int
- ECMWF CCI2 endpoint:
host_base = object-store.os-api.cci2.ecmwf.int
- ECMWF CCI1 endpoint:
EUMETSAT endpoint: host_base =
s3.waw3-1.cloudferro.com
Fill in the . You will need to replace
<provider>
by eitherecmwf
oreumetsat
(your storage provider), as well as<youraccesskey>
and<yoursecretkey>
that will be given to you by the providerNo Format host_base = storage.<provider>.europeanweather.cloud<EUM or ECMWF endpoint> host_bucket = access_key = <youraccesskey> secret_key = <yoursecretkey> use_https = True
Warning July 2021: EUMETSAT's S3 provision has an incomplete SSL certificate; you can either set use_https
toFalse
or temporarily use the underlying service'shost_base
ofobs.eum.otc.icsi.eumetsat.int
(the infrastructure for 2022 is in ITT, and we are waiting to see where it ends up before taking the cost of replacing the certificate)# Needed for EUMETSAT, as the provider currently supplies a "CloudFerro" SSL certificate. Skip if ECMWF. check_ssl_certificate = False
Basic tasks
If you type s3cmd -h
you will see the different options of the command, but here are the basics:
List buckets
No Format |
---|
s3cmd ls |
Create a bucket
No Format |
---|
s3cmd mb s3://yourbucket |
List bucket contents
No Format |
---|
s3cmd ls s3://yourbucket |
Get data from bucket
No Format |
---|
s3cmd get s3://newbucket/file.txt |
Put data into bucket
No Format |
---|
s3cmd put file.txt s3://newbucket/ |
Remove data from bucket
No Format |
---|
s3cmd rm s3://newbucket/file.txt |
Remove empty bucket
No Format |
---|
s3cmd rb s3://yourbucket/ |
Configure automatic expiry of data
No Format |
---|
s3cmd expire --expiry-days=14 s3://yourbucket/ |
Information about a bucket
No Format |
---|
s3cmd info s3://newbucket |
Remove automatic expiry policy
No Format |
---|
s3cmd dellifecycle s3://yourbucket/ |
Mounting your bucket with S3FS via FUSE
You may also mount your bucket to expose the files in your S3 bucket as if they were on a local disk. Generally S3 cannot offer the same performance or semantics as a local file system, but it can be useful for legacy applications that mainly need to read data and expect the files to be in a conventional file system. You can find more information here.
S3FS installation
First of all, make sure you have S3FS installed in your VM. On CentOS:
...
No Format |
---|
sudo apt install s3fs |
Configure S3FS
You need to store your credentials in a file so S3FS can authenticate with the service. You need to replace <youraccesskey>
and <yoursecretkey>
by your actual credentials.
No Format |
---|
echo <youraccesskey>:<yoursecretkey> | sudo tee /root/.passwd-s3fs sudo chmod 600 /root/.passwd-s3fs |
Setting up an automatic mount
Assuming you want to mount your bucket in /mnt/yourbucket
, here is what you need to do:
No Format |
---|
sudo mkdir /mnt/yourbucket echo "s3fs#yourbucket /mnt/yourbucket fuse _netdev,allow_other,nodev,nosuid,uid=$(id -u),gid=$(id -g),use_path_request_style,url=https://storage.<provider>.europeanweather.cloud<s3_endpoint> 0 0" | sudo tee -a /etc/fstab sudo mount -a |
Again, you must replace <provider>
<s3_endpoint>
by the ecmwf
or eumetsat
, and relevant endpoint at ECMWF or EUMETSAT, and you may customise other mount options if you wish to do so. At this point you should have your bucket mounted and ready to use.
Related articles
Content by Label | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...