EUMETSAT is establishing has established a new set of data access services that will bring are bringing the full data catalog catalogue online.  Some more General information can be found here (in particular, see the part on the Data Store): https://wwwuser.eumetsat.int/website/home/Data/DataDelivery/NewPilotDataServices/index.html

At the initial opening of EWC to users, these services were not yet fully tested, populated or widely-accessible.  Even once the services are open, it will still take time to fully populate them.

As a workaround for this intermediate period, virtual machines on the EUMETSAT cloud are able to mount an NFS share holding selected datasets.  This server will be maintained until it is no longer needed.

Mounting the EUMETSAT NFS data archive

Setting up your instance

In order to mount the NFS drive, first, you need to create the Virtual Machine instances in the Morpheus UI. You can follow the Provision a new instance - web Knowledge Base Article to get started.

Mounting NFS Share

After logging in to the virtual machine that you want to access the EUMETSAT data, all you need is to do is mount the NFS drive that rests on 198.18.255.11 onto the directory where you want it.  You will need sudo permissions or root access to accomplish the major steps here.

Verify access to the NFS server

You can see all exported file systems on the EUMETSAT NFS server without mounting it using the showmount command like in the example below:

karatosun@myfirstvm ~$ showmount --exports 198.18.255.11
Export list for 198.18.255.11:
...  (lot of text, but should include)
/ifs/data/EUMETSAT_DATA 
/ifs/data/EUMETCAST 
...

 Create the (empty) directory where you'll mount the NFS share

You can place this anywhere you wish; we have chosen /eumetsatdata in the example here

karatosun@myfirstvm ~$ sudo mkdir /eumetsatdata
karatosun@myfirstvm ~$ sudo echo "run sudo mount -a" > /eumetsatdata/NFS_NOT_MOUNTED  # put a placeholder file in the directory to make it clear what's (not) happened
karatosun@myfirstvm ~$ sudo chmod a+rX,go-w /eumetsatdata                             # minimise write permissions to prevent mistakes if the mount isn't there

data-access.

In addition, some selected datasets can be accessed from EWC VMs from the EWC object storage. Please contact us, if you need additional data which is not provided in the Data Store, or you have special conditions for access. 

...

Using EUMETSAT data buckets

Available EUMETSAT data buckets are listed below with the corresponding mount point in case of using ready workflow. The buckets are available from the whole EWC (including ECMWF side) without credentials.

EUMETSAT Data Buckets:

Regarding the data, please refer the EUMETNET documentation

Following GeoRing source data are available in the EWC data buckets. Please note that these are not included in the workflow to mount data via FUSE mount.

Using the data directly from workloads 

It is advisable to use the data directly from workloads (scripts / applications). Please consult e.g.

• s3 using Python libraries
• Object storage: How to use s3cmd and s3fs
• s3 from command line

for more help. 

...

Mounting the buckets using FUSE

Following describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using python.  Whenever possible, users are advised to use the buckets directly from the workloads.

Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.

Using the Workflow

1. Go to instances and select the instance of interest
   Image Added
2. Select Actions → Run Workflow
   Image Added
   
3. Select Mount EUMETSAT databuckets from Workflow drop-down meny and click Execute. Command options are not needed.
   Image Added
   

You can then follow the workflow from the output or by going to Instance History tab. After the workflow is executed, you can access the data at /eumetsatdata.

Manual mount

Following steps are using ASCAT data as an example. Please modify the commands to mount any other bucket. 

1. Make sure you have latest rclone installed

   Code Block
   curl https://rclone.org/install.sh | sudo bash

   
   

2. Create mount directory, i.e. for ascat:

   Code Block
   sudo mkdir -p /eumetsatdata/ascat && sudo chown -R $USER:$USER /eumetsatdata/ascat && chmod 755 /eumetsatdata/ascat

   
   

3. Put rclone like listed below config into a config file. For example, into /etc/eum_bucket_creds (keys without brackets) 

   Code Block
   sudo sh -c "cat >>/etc/eum_bucket_creds <<-EOF [eumetsat-ascat] type = s3 provider = Ceph endpoint = https://s3.waw3-1.cloudferro.com EOF

   
   

4. Now you can already mount the data with command:

   Code Block
   sudo rclone --config /etc/eum_bucket_creds mount --allow-other --daemon eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat

Setting up a service

To have mount points available all the time and after reboots, one need to set up the service and add mount points into fstab.

1. Link rclone

   Code Block
   sudo ln -s /usr/bin/rclone /sbin/mount.rclone

   
   

2. Create systemd service:, for example for ascat:

   Code Block

   sudo sh -c "cat >>/etc/systemd/system/eum-data-ascat.service" <<-EOF [Unit] Description=Mount EUMETSAT data bucket ascat After=network-online.target [Mount] Type=rclone What=eumetsat-ascat:ascat.metop-abc.level-1b.native Where=/eumetsatdata/ascat Options=ro,allow_other,args2env,vfs-cache-mode=writes,config=/etc/eum_bucket_creds,cache-dir=/var/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m' EOF

   
   

3. Add corresponding line into /etc/fstab:

   Code Block
   echo "eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat rclone ro,noauto,nofail,x-systemd.automount,args2env,allow_other,vfs_cache_mode=writes,config=/etc/eum_bucket_creds,cache_dir=/var/cache/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m 0 0" | sudo tee -a /etc/fstab

   
   

4. Mount the folder:

   Code Block
   sudo mount /eumetsatdata/ascat

Mount NFS drive to the mount point

This command will do a once-off mount.   If it works, you may wish to make it permanent as described in step 5

karatosun@myfirstvm ~$ sudo mount 198.18.255.11:/ifs/data/EUMETSAT_DATA /eumetsatdata

If step 3 worked, you can now navigate to the /eumetsatdata directory and list all the available data with the ls -lah command

[karatosun@myfirstvm ~]$ cd /eumetsatdata/
[karatosun@myfirstvm eumetsatdata]$ ls -lah
total 192K
drwxrwxrwx.  7 nobody   nobody   124 May 26 13:00 .
dr-xr-xr-x. 19 root     root     260 Jul  6 13:00 ..
drwxrwsr-x.  5     1002     1002 101 Nov  7  2018 eumetcast
drwxr-xr-x.  3 nobody   nobody    24 Apr 23  2018 mviri
drwxr-xr-x.  4 nobody   nobody    52 May 28 17:26 seviri

Making the NFS mount permanent and persistent across reboots (optional)

Edit /etc/fstab file of the VM with your editor of choice

karatosun@myfirstvm ~$ sudo nano /etc/fstab

and add the following line

198.18.255.11:/ifs/data/EUMETSAT_DATA /eumetsatdata nfs defaults 0 0

A complete example of a /etc/fstab can be seen in the block below (do not copy all of this!)

# /etc/fstab
# Created by anaconda on Thu Aug 8 12:37:33 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=3ef2b806-efd7-4eef-aaa2 / xfs defaults 0 0

# mount the EUMETSAT NFS server to /eumetsatdata
198.18.255.11:/ifs/data/EUMETSAT_DATA /eumetsatdata nfs defaults 0 0

To test the persistent configuration, unmount the share, then try to automatically remount it:

karatosun@myfirstvm ~$ sudo umount /eumetsatdata
karatosun@myfirstvm ~$ sudo mount -a

Troubleshooting

...

Limitations

1. Access to the archive is read-only.  This is a technical limitation due to the way the NFS server is mapped into the VM's address space.
2. Only a limited subset of EUMETSAT data is available.   At the time of writing, this included Meteosat L1.5 data, a cloud type dataset and a few example files from EUMETCAST.  Please contact us if you need something that is not currently there.

Related articles

...