...
In addition, some selected datasets can be accessed from EWC VMs from the EWC object storage. Please contact us, if you need additional data which is not provided in the Data Store, or you have special conditions for access.
Table of Contents |
---|
...
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Primary method: using EUMETSAT data servicesTo avoid duplication, the information on how to use the new data services can be found on the following links:
In the future, it may become possible to directly access the object storage behind the Data Store for applications that need very high throughput (e.g. mass data processing). Please contact us if you have tried the existing possibilities and believe you need this capability. |
...
...
Using EUMETSAT data buckets
Available EUMETSAT data buckets are listed below with the corresponding mount point in case of using ready workflow. The buckets are available from the whole EWC (including ECMWF side) without credentials.
Alert | ||
---|---|---|
| ||
Please note that all buckets are updated on best-effort basis and should not be used for operational workloads. For operational usage, we recommend to use services provided by the original data providers. |
EUMETSAT Data Buckets:
Bucket | Mount point | Time range | ||
---|---|---|---|---|
ascat.metop-abc.level-1b.native | /eumetsatdata/ascat | 2012-10-22T11:00:00Z - 2013-04-24T08:57:00Z | ||
cmsaf-cloud-type.meteosat-0-degree.level-2.netcdf-4 | /eumetsatdata/cloud | 2004-2017 | ||
geo-hsr-simulation.0-degree.level-15.netcdf-4 | /eumetsatdata/geo-hsr-test-data | |||
iasi.metop-a.level-1c.native | /eumetsatdata/iasi | 2007-2020 | ||
mviri.meteosat-0-degree.level-15.native | /eumetsatdata/mviri | 1981-2017 | ||
nwc-saf.0-degree.level-2 | /eumetsatdata/nwcsaf | 2019-2021 | ||
nwc-saf.0-degree.level-2-{year} | /eumetsatdata/nwcsaf-2022 | 2022 | ||
- current | ||||
nwc-saf.0-degree.level-2-cf | /eumetsatdata/nwcsaf-cf | 2019-2020 | ||
seviri.meteosat-0-degree.level-15.native | /eumetsatdata/seviri | 2020 | ||
drought-and-vegetation.prototype-datacube.netcdf | /eumetsatdata/psdc | |||
era5.ecmwf.analyses-eo-subset.grib
| - | 2005/01 - 2024/02 |
Footnotes Display |
---|
Anchor | ||||
---|---|---|---|---|
|
Regarding the data, please refer the EUMETNET documentation
rainfall_accumulation.composite.opera.bufr | /eumetsatdata/opera-rainfall-accumulation-bufr | 15 Dec 2010 - current |
rainfall_accumulation.composite.opera.hdf5 | /eumetsatdata/opera-rainfall-accumulation-hdf5 | 4 Sep 2012 - current |
rainfall_rate.composite.opera.bufr | /eumetsatdata/opera-rainfall-rate-bufr | 15 Dec 2010 - current |
rainfall_rate.composite.opera.hdf5 | /eumetsatdata/opera-rainfall-rate-hdf5 | 4 Sep 2012 - current |
reflectivity.composite.opera.bufr | /eumetsatdata/opera-reflectivity-bufr | 15 Dec 2010 - current |
reflectivity.composite.opera.hdf5 | /eumetsatdata/opera-reflectivity-hdf5 | 4 Sep 2012 - current |
Anchor | ||||
---|---|---|---|---|
|
Following GeoRing source data are available in the EWC data buckets. Please note that these are not included in the workflow to mount data via FUSE mount.
Dataset | Format | Bucket | Details |
Himawari | NetCDF 4 | s3://rep.ahi-jma-nrt-fulldisk.ahixxx10.himawari8.level10.netcdf | Currently 3 days from each month since 2015. To be populated with all data since 2015 |
GOES-GVAR | Area | s3://rep.imager_noaa_class.gvar_img.goes-[08-15].l0.area | All available data copied to EWC |
GOES-PreGVAR | Area | s3://rep.vissr_noaa_class.pre_gvar.goes-[01-07].l0.native | All available data copied to EWC |
SMS | Area | s3://rep.vissr_noaa_class.pre_gvar.sms-[01-02].l0.native | All available data copied to EWC |
Using the data directly from workloads
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Recommended way |
It is advisable to use the data directly from workloads (scripts / applications). Please consult e.g.
for more help.
...
Mounting the buckets using FUSE
Info |
---|
For data access, you will need to provision your instance on the EUMETSAT cloud. You can follow the Provision a new instance - web Knowledge Base Article to get started. |
...
Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.
Using the Workflow
- Go to instances and select the instance of interest
- Select Actions → Run Workflow
- Select Mount EUMETSAT databuckets from Workflow drop-down meny and click Execute. Command options are not needed.
You can then follow the workflow from the output or by going to Instance History tab. After the workflow is executed, you can access the data at /eumetsatdata.
Manual mount
Following steps are using ASCAT data as an example. Please modify the commands to mount any other bucket.
Make sure you have latest rclone installed
Code Block curl https://rclone.org/install.sh | sudo bash
Create mount directory, i.e. for ascat:
Code Block sudo mkdir -p /eumetsatdata/ascat && sudo chown -R $USER:$USER /eumetsatdata/ascat && chmod 755 /eumetsatdata/ascat
Put rclone like listed below config into a config file. For example, into
/etc/eum_bucket_creds (keys without brackets)
Code Block sudo sh -c "cat >>/etc/eum_bucket_creds <<-EOF [eumetsat-ascat] type = s3 provider = Ceph endpoint = https://s3.waw3-1.cloudferro.com EOF
Now you can already mount the data with command:
Code Block sudo rclone --config /etc/eum_bucket_creds mount --allow-other --daemon eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat
Setting up a service
To have mount points available all the time and after reboots, one need to set up the service and add mount points into fstab.
Link rclone
Code Block sudo ln -s /usr/bin/rclone /sbin/mount.rclone
Create systemd service:, for example for ascat:
Code Block sudo sh -c "cat >>/etc/systemd/system/eum-data-ascat.service" <<-EOF [Unit] Description=Mount EUMETSAT data bucket ascat After=network-online.target [Mount] Type=rclone What=eumetsat-ascat:ascat.metop-abc.level-1b.native Where=/eumetsatdata/ascat Options=ro,allow_other,args2env,vfs-cache-mode=writes,config=/etc/eum_bucket_creds,cache-dir=/var/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m' EOF
Add corresponding line into /etc/fstab:
Code Block echo "eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat rclone ro,noauto,nofail,x-systemd.automount,args2env,allow_other,vfs_cache_mode=writes,config=/etc/eum_bucket_creds,cache_dir=/var/cache/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m 0 0" | sudo tee -a /etc/fstab
Mount the folder:
Code Block sudo mount /eumetsatdata/ascat
Limitations
- Access to the archive is read-only.
- Only a limited subset of EUMETSAT data is available. Please contact us if you need something that is not currently there.
Related articles
Content by Label | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...