You may want to make use of Object Storage in your infrastructure. An S3-compatible service can be enabled for your tenant so you can store or retrieve data from buckets stored in the cloud, offered by both ECMWF and EUMETSAT.
At the moment the access to this service is not activated by default for every tenant. If you wish to use it, please raise an issue through the Support Portal requesting access to this service.
You may also use this guide to use any other S3 storage service such as AWS from your instances at the European Weather Cloud. Just adapt the host and credential information accordingly.
Managing your Object Storage with S3cmd
S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol.
Many other advanced tools (e.g. https://rclone.org/) exist, as do APIs for many languages, but this article aims only to demonstrate the basics.
Install the tool
The easiest is to install it through the system package manager
Or for Ubuntu:
Alternatively, you may get the latest version from PyPi
You will need to configure
s3cmd before you can use it. The tool will read the configuration on
Create the configuration file if it does not exist:
Edit the file and set up at least the following parameters.
- ECMWF CCI1 endpoint:
host_base = object-store.os-api.cci1.ecmwf.int
- ECMWF CCI2 endpoint:
host_base = object-store.os-api.cci2.ecmwf.int
- ECMWF CCI1 endpoint:
Fill in the
<yoursecretkey>that will be given to you by the provider
If you type
s3cmd -h you will see the different options of the command, but here are the basics:
Create a bucket
List bucket contents
Get data from bucket
Put data into bucket
Remove data from bucket
Remove empty bucket
Configure automatic expiry of data
Information about a bucket
Remove automatic expiry policy
Mounting your bucket with S3FS via FUSE
You may also mount your bucket to expose the files in your S3 bucket as if they were on a local disk. Generally S3 cannot offer the same performance or semantics as a local file system, but it can be useful for legacy applications that mainly need to read data and expect the files to be in a conventional file system.
First of all, make sure you have S3FS installed in your VM. On CentOS:
You need to store your credentials in a file so S3FS can authenticate with the service. You need to replace
<yoursecretkey> by your actual credentials.
Setting up an automatic mount
Assuming you want to mount your bucket in
/mnt/yourbucket, here is what you need to do:
Again, you must replace
<s3_endpoint> by the relevant endpoint at ECMWF or EUMETSAT, and you may customise other mount options if you wish to do so. At this point you should have your bucket mounted and ready to use.