Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

EUMETSAT is establishing has established a new set of data access services that will bring are bringing the full data catalog catalogue online.  Some more General information can be found here:  https://wwwuser.eumetsat.int/data-access-our-data.Knowledge bases for each of the new data services can be found through this landing page: https://eumetsatspace.atlassian.net/wiki/spaces/EUM/overview

These services are provided on a best-effort basis. Even now the services are open, it will still take time to fully populate them. In addition, some selected datasets can be accessed from EWC VMs from the EWC object storage. Please contact us, if you need an additional data which is not provided in the Data Store, or you have special conditions for access

...

Table of Contents

Panel
borderColorgreen
borderWidth2
borderStylesolid

Primary method: using EUMETSAT data services

To avoid duplication, the information on how to use the new data services can be found on the following links:

...

In the future, it may become possible to directly access the object storage behind the Data Store for applications that need very high throughput (e.g. mass data processing).  Please contact us if you have tried the existing possibilities and believe you need this capability.

...


...

Using

...

Setting up your instance

Info

For data access, you will need to provision your instance on the EUMETSAT cloud (or establish a VPN connection to the EUMETSAT cloud).

In order to mount the buckets as filesystem, you need to create the Virtual Machine instances in the Morpheus UI. You can follow the Provision a new instance - web Knowledge Base Article to get started.

This article describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using pythonWhenever possible, users are advised to use the buckets directly from the workloads.

...

EUMETSAT

...

data buckets

...

Available EUMETSAT data buckets are listed below with the corresponding mount point in case of using ready workflow. You can get access keys for the data buckets from support. The buckets are available from the whole EWC (including ECMWF side) without credentials.

Alert
typeWarning

Please note that all buckets are updated on best-effort basis and should not be used for operational workloads. For operational usage, we recommend to use services provided by the original data providers.


EUMETSAT Data Buckets:

BucketMount pointTime range
ascat.metop-abc.level-1b.native/eumetsatdata/ascat2012-10-22T11:00:00Z - 2013-04-24T08:57:00Z
cmsaf-cloud-type.meteosat-0-degree.level-2.netcdf-4/eumetsatdata/cloud2004-2017
geo-hsr-simulation.0-degree.level-15.netcdf-4/eumetsatdata/geo-hsr-test-data
iasi.metop-a.level-1c.native/eumetsatdata/iasi2007-20162020
mviri.meteosat-0-degree.level-15.native/eumetsatdata/mviri1981-2017
nwc-saf.0-degree.level-2/eumetsatdata/nwcsaf2019-2021
nwc-saf.0-degree.level-2-{year}/eumetsatdata/nwcsaf-20222022 onwards- current
nwc-saf.0-degree.level-2-cf/eumetsatdata/nwcsaf-cf2019-2020
seviri.meteosat-0-degree.level-15.native/eumetsatdata/seviri2020
drought-and-vegetation.prototype-datacube.netcdf/eumetsatdata/psdc

Anchor
OPERA
OPERA
OPERA data:

Regarding the data, please refer the EUMETNET documentation

rainfall_accumulation.composite.opera.bufr/eumetsatdata/opera-rainfall-accumulation-bufr15 Dec 2010 - current
rainfall_accumulation.composite.opera.hdf5/eumetsatdata/opera-rainfall-accumulation-hdf54 Sep 2012 - current
rainfall_rate.composite.opera.bufr/eumetsatdata/opera-rainfall-rate-bufr15 Dec 2010 - current
rainfall_rate.composite.opera.hdf5/eumetsatdata/opera-rainfall-rate-hdf54 Sep 2012 - current
reflectivity.composite.opera.bufr/eumetsatdata/opera-reflectivity-bufr15 Dec 2010 - current
reflectivity.composite.opera.hdf5/eumetsatdata/opera-reflectivity-hdf54 Sep 2012 - current

Using the data directly from workloads 

Panel
borderColorgreen
borderWidth2
borderStylesolid

Recommended way

It is advisable to use the data directly from workloads (scripts / applications). Please consult e.g.

for more help. 


...

Mounting the buckets using FUSE

Info

For data access, you will need to provision your instance on the EUMETSAT cloud. You can follow the Provision a new instance - web Knowledge Base Article to get started.

Following describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using pythonWhenever possible, users are advised to use the buckets directly from the workloads.

Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.

Using the Workflow

  1. Go to instances and select the instance of interest
  2. Select Actions → Run Workflow

  3. Select Mount EUMETSAT databuckets from Workflow drop-down meny and click Execute. Command options are not needed.

You can then follow the workflow from the output or by going to Instance History tab. After the workflow is executed, you can access the data at /eumetsatdata.

Manual mount

Following steps are using ASCAT data as an example. Please modify the commands to mount any other bucket. 

  1. Make sure you have latest rclone installed

    Code Block
    curl https://rclone.org/install.sh | sudo bash


  2. Create mount directory, i.e. for ascat:

    Code Block
    sudo mkdir -p /eumetsatdata/ascat && sudo chown -R $USER:$USER /eumetsatdata/ascat && chmod 755 /eumetsatdata/ascat


  3. Put rclone like listed below config into a config file. For example, into /etc/eum_bucket_creds (keys without brackets) 

    Code Block
    sudo sh -c "cat >>/etc/eum_bucket_creds <<-EOF
    [eumetsat-ascat]
    type = s3
    provider = Ceph
    endpoint = https://s3.waw3-1.cloudferro.com
    EOF


  4. Now you can already mount the data with command:

    Code Block
    sudo rclone --config /etc/eum_bucket_creds mount --allow-other --daemon eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat

Setting up a service

To have mount points available all the time and after reboots, one need to set up the service and add mount points into fstab.

  1. Link rclone

    Code Block
    sudo ln -s /usr/bin/rclone /sbin/mount.rclone


  2. Create systemd service:, for example for ascat:

    Code Block
    sudo sh -c "cat >>/etc/systemd/system/eum-data-ascat.service" <<-EOF
    [Unit]
    Description=Mount EUMETSAT data bucket ascat
    After=network-online.target
    
    [Mount]
    Type=rclone
    What=eumetsat-ascat:ascat.metop-abc.level-1b.native
    Where=/eumetsatdata/ascat
    Options=ro,allow_other,args2env,vfs-cache-mode=writes,config=/etc/eum_bucket_creds,cache-dir=/var/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m'
    EOF


  3. Add corresponding line into /etc/fstab:

    Code Block
    echo "eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat rclone ro,noauto,nofail,x-systemd.automount,args2env,allow_other,vfs_cache_mode=writes,config=/etc/eum_bucket_creds,cache_dir=/var/cache/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m 0 0" | sudo tee -a /etc/fstab


  4. Mount the folder:

    Code Block
    sudo mount /eumetsatdata/ascat

Limitations

  1. Access to the archive is read-only.
  2. Only a limited subset of EUMETSAT data is available. Please contact us if you need something that is not currently there.

Content by Label
showLabelsfalse
max5
spacesEWCLOUDKB
showSpacefalse
sortmodified
reversetrue
typepage
excludeCurrenttrue
cqllabel in ("data","kb-how-to-article","s3") and type = "page" and space = "EWCLOUDKB"
labelsdata kb-how-to-article

...