...
As a workaround for this intermediate period, virtual machines on the EUMETSAT cloud are able to mount an NFS share use object storage data buckets holding selected datasets. This server will be maintained until it is no longer needed.
Option 1: Using EUMETSAT data services
...
In the future, it may become possible to directly access the object storage behind the Data Store for applications that need very high throughput (e.g. mass data processing). Please contact us if you have tried the existing possibilities and believe you need this capability.
Option 2:
...
Using EUMETSAT
...
data
...
buckets
Setting up your instance
Info |
---|
For data access, you will need to provision your instance on the EUMETSAT cloud (or establish a VPN connection to the EUMETSAT cloud). |
In order to mount the NFS drive, firstbuckets as filesystem, you need to create the Virtual Machine instances in the Morpheus UI. You can follow the the Provision a new instance - web Knowledge Base Article to get started.
Mounting NFS Share
After logging in to the virtual machine that you want to access the EUMETSAT data, all you need is to do is mount the NFS drive that rests on 198.18.255.33 onto the directory where you want it. You will need sudo permissions or root access to accomplish the major steps here.
Verify access to the NFS server
You can see all exported file systems on the EUMETSAT NFS server without mounting it using the showmount command like in the example below:
Code Block | ||
---|---|---|
| ||
karatosun@myfirstvm ~$ showmount --exports 198.18.255.33
Export list for 198.18.255.33:
... (lot of text, but should include)
/ifs/data/EUMETSAT_DATA
/ifs/data/EUMETCAST
...
|
Info |
---|
If you receive an error (e.g. timeout), please refer to the troubleshooting section below. |
Create the (empty) directory where you'll mount the NFS share
You can place this anywhere you wish; we have chosen /eumetsatdata in the example here
Code Block | ||
---|---|---|
| ||
karatosun@myfirstvm ~$ sudo mkdir /eumetsatdata
karatosun@myfirstvm ~$ sudo echo "run sudo mount -a" > /eumetsatdata/NFS_NOT_MOUNTED # put a placeholder file in the directory to make it clear what's (not) happened
karatosun@myfirstvm ~$ sudo chmod a+rX,go-w /eumetsatdata # minimise write permissions to prevent mistakes if the mount isn't there |
Info |
---|
The last two commands add a little extra safety for the case that the directory is unexpectedly not mounted |
Mount NFS drive to the mount point
This command will do a once-off mount. If it works, you may wish to make it permanent as described in step 4
Code Block | ||
---|---|---|
| ||
karatosun@myfirstvm ~$ sudo mount 198.18.255.33:/ifs/data/EUMETSAT_DATA /eumetsatdata |
Info |
---|
If you receive an error (e.g. timeout), please refer to the troubleshooting section below. |
If step 3 worked, you can now navigate to the /eumetsatdata directory and list all the available data with the ls -lah command
Code Block | ||
---|---|---|
| ||
[karatosun@myfirstvm ~]$ cd /eumetsatdata/
[karatosun@myfirstvm eumetsatdata]$ ls -lah
total 192K
drwxrwxrwx. 7 nobody nobody 124 May 26 13:00 .
dr-xr-xr-x. 19 root root 260 Jul 6 13:00 ..
drwxrwsr-x. 5 1002 1002 101 Nov 7 2018 eumetcast
drwxr-xr-x. 3 nobody nobody 24 Apr 23 2018 mviri
drwxr-xr-x. 4 nobody nobody 52 May 28 17:26 seviri |
Info |
---|
Note that the ownerships and permissions will generally be "nobody" |
This article describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using python. Whenever possible, users are advised to use the buckets directly from the workloads.
Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.
Available EUMETSAT data buckets are listed below with the corresponding mount point in case of using ready workflow. You can get access keys for the data buckets from support.
EUMETSAT Data Buckets:
Bucket | Mount point |
---|---|
ascat.metop-abc.level-1b.native | /eumetsatdata/ascat |
cmsaf-cloud-type.meteosat-0-degree.level-2.netcdf-4 | /eumetsatdata/cloud |
geo-hsr-simulation.0-degree.level-15.netcdf-4 | /eumetsatdata/geo-hsr-test-data |
iasi.metop-a.level-1c.native | /eumetsatdata/iasi |
mviri.meteosat-0-degree.level-15.native | /eumetsatdata/mviri |
nwc-saf.0-degree.level-2 | /eumetsatdata/nwcsaf |
nwc-saf.0-degree.level-2-cf | /eumetsatdata/nwcsaf-cf |
seviri.meteosat-0-degree.level-15.native | /eumetsatdata/seviri |
drought-and-vegetation.prototype-datacube.netcdf | /eumetsatdata/psdc |
Using the Workflow
- Go to instances
- Select the instance of interest
- Select Actions → Run Workflow
- Select Mount EUMETSAT databuckets from Workflow drop-down meny and click Execute. Command options are not needed.
You can then follow the workflow from the output or by going to Instance History tab. After the workflow is executed, you can access the data at /eumetsatdata.
Manual mount
Following steps are using ASCAT data as an example. Please modify the commands to mount any other bucket.
Make sure you have latest rclone installed
Code Block curl https://rclone.org/install.sh | sudo bash
Create mount directory, i.e. for ascat:
Code Block sudo mkdir -p /eumetsatdata/ascat && sudo chown -R $USER:$USER /eumetsatdata/ascat && chmod 755 /eumetsatdata/ascat
Put rclone like listed below config into a config file. For example, into
/etc/eum_bucket_creds
Code Block sudo sh -c "cat >>/etc/eum_bucket_creds <<-EOF [eumetsat-ascat] type = s3 provider = Ceph env_auth = false access_key_id={{ aws_access_key_id }} secret_access_key={{ aws_secret_access_key }} endpoint = https://s3.waw3-1.cloudferro.com EOF
Now you can already mount the data with command:
Code Block sudo rclone --config /etc/eum_bucket_creds mount eumetsat-ascat:847632567fbf45408555318627075a26:ascat.metop-abc.level-1b.native /eumetsatdata/ascat
Please note the special naming convention rclone_config_name:ceph_tenancy_id:bucket_name. This is due to the CEPH multi-tenancy setup. If you are mounting your own bucket, you can drop the tenancy id and use the standard bucket naming style: rclone_config_name:bucket_name.
Setting up a service
To have mount points available all the time and after reboots, one need to set up the service and add mount points into fstab.
Link rclone
Code Block sudo ln -s /usr/bin/rclone /sbin/mount.rclone
Create systemd service:, for example for ascat:
Code Block sudo sh -c "cat >>/etc/systemd/system/eum-data-ascat.service" <<-EOF [Unit] Description=Mount EUMETSAT data bucket ascat After=network-online.target [Mount] Type=rclone What=eumetsat-ascat:847632567fbf45408555318627075a26:ascat.metop-abc.level-1b.native Where=/eumetsatdata/ascat Options=ro,allow_other,args2env,vfs-cache-mode=writes,config=/etc/eum_bucket_creds,cache-dir=/var/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m' EOF
Add corresponding line into /etc/fstab:
Code Block echo "eumetsat-ascat:847632567fbf45408555318627075a26:ascat.metop-abc.level-1b.native /eumetsatdata/ascat rclone ro,noauto,nofail,x-systemd.automount,args2env,allow_other,vfs_cache_mode=writes,config=/etc/eum_bucket_creds,cache_dir=/var/cache/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m 0 0" | sudo tee -a /etc/fstab
Mount the folder:
Code Block sudo mount /eumetsatdata/ascat
Making the NFS mount permanent and persistent across reboots (optional)
Edit /etc/fstab file of the VM with your editor of choice
Code Block | ||
---|---|---|
| ||
karatosun@myfirstvm ~$ sudo nano /etc/fstab |
and add the following line
Code Block | ||
---|---|---|
| ||
198.18.255.33:/ifs/data/EUMETSAT_DATA /eumetsatdata nfs defaults 0 0 |
A complete example of a /etc/fstab can be seen in the block below (do not copy all of this!)
Code Block | ||||
---|---|---|---|---|
| ||||
# /etc/fstab
# Created by anaconda on Thu Aug 8 12:37:33 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=3ef2b806-efd7-4eef-aaa2 / xfs defaults 0 0
# mount the EUMETSAT NFS server to /eumetsatdata
198.18.255.11:/ifs/data/EUMETSAT_DATA /eumetsatdata nfs defaults 0 0 |
To test the persistent configuration, unmount the share, then try to automatically remount it:
Code Block | ||
---|---|---|
| ||
karatosun@myfirstvm ~$ sudo umount /eumetsatdata
karatosun@myfirstvm ~$ sudo mount -a
|
Info |
---|
If you see an error and the manual mount worked previously, please check for typos! |
Troubleshooting
Warning |
---|
Like all NFS mounts, if there is a problem accessing the server (networking issue, firewall, server down, server overloaded), this will cause attempts to access the mounted directory to freeze the process until the server returns. |
...
Limitations
- Access to the archive is read-only. This is a technical limitation due to the way the NFS server is mapped into the VM's address space.
- Only a limited subset of EUMETSAT data is available. At the time of writing, this included Meteosat L1. 5 data, a cloud type dataset and a few example files from EUMETCAST. Please contact us if you need something that is not currently there.
...
Content by Label | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
- RClone: https://rclone.org/
Page properties | ||
---|---|---|
| ||
|
...