Versions Compared

Old Version 4

changes.mady.by.user Yigit Oner Altintas

Saved on

compared with

New Version Current

changes.mady.by.user Yigit Oner Altintas

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction

On In the EUMETSAT side section of the European Weather Cloud, we have there's now an instance that you can provision to install equipped with the Data Tailor Standalone with all and EUMETSAT Data Access Client (EUMDAC) together, which includes all the currently available plugins on a Rocky Linux system.  Below is , you'll find a small guide showing the installation and that covers the provisioning, a minimal post-installation process, and usage instructions. 

The Data Tailor is a product customization toolbox designed to allow enable users to modify tailor satellite data according to their specific needs. It provides offers the capabilities ability to subset and aggregate data products in across space and time, filter layers, generate quick looks, re-project reproject data onto new coordinate reference systems, and reformat data into common widely used Geographic Information System (GIS) formats such as netCDF and GeoTIFF and , as well as image formats such as like JPEG and PNG. The Data Tailor can currently can customise customize data from various satellite collections of , including METOP, MSG, and Sentinel-3 satellites. There are two types versions of the Data Tailor. The : the Data Tailor Web Service (DTWS) and the Data Tailor Standalone. The DTWS allows users to perform customizations on customize products in the EUMETSAT Data Store before they are downloaded. The downloading them. On the other hand, the Data Tailor Standalone on the other hand is an application that you can install to be installed on your machine and customise , enabling you to customize downloaded products. Comparing to Unlike the Data Tailor Web Services that you can use with  EUMDAC or through , which can be used with EUMDAC or via https://tailor.eumetsat.int, the Data Tailor standalone version is generally faster and don't does not have the limitations like such as maximum concurrent jobs or workspace size.  For For more information on capabilities of the Data Tailor, please refer to Data Tailor Standalone Guide on User Portal and for more information about the available products and customisations inside the Data Tailor, please go to Products and Customisations Available in the Data Tailor page.

Table of Contents

Current limitations

...

Warning
  • Due to design of the software, only one linux user is able to use the software without

...

  • additional settings. When a user runs the application for the first time, various configuration files are created with permissions only editable by that user.

Installation

Provisioning

Image Removed

...

  1. Create the Data Tailor standalone instance by clicking on + ADD button on your Morpheus Dashboard.

...

  1. Search for Data Tailor Standalone, select it and click next.
    Image Added
  2. Choose a NAME for the instance type and move to next step.
    Image Added
  3.  Insert at least the following inputs and click next:
    • layout: make sure it is rocky-datatailor
    • plan: select one with at least 16GB of RAM (e.g. eo2.xlarge)
    • network: private
    • security group: ssh
      Image Added
  4. Once done, click Complete and the provisioning will start.
  5. Look for the instance list (Provisioning → Instances) to check the status of the instance.
  6. Once ready you will see the green Running sign below the instance as below:

  1. Image Modified

Post-provisioning

  1. SSH into your instance following the Remote Access to the VM

...

  1. documentation. Alternatively, you can also use the console on Morpheus from your instance (Provisioning → Instances → select your instance and go to the Console section)
  2. Running the following commands you can verify if the installation was successful.
    Code Block
    languagebash
    titleCheck conda installation and

...

  1. epct-desktop environment
    $ conda env list

...

  1. 
# conda environments:

...

  1. 
#

...

  1. 
base                  *  /usr/local/miniconda

...

  1. 
epct-desktop             /usr/local/miniconda/envs/

...

  1. epct-desktop
  2. After confirming the installation, add conda environment to your shell and then restart it.
    Code Block
    languagebash
    titleIntegrate conda in your shell and restart your shell

...

  1. $ conda init

...

  1. 
no change     /usr/local/miniconda/condabin/conda

...

  1. 
no change     /usr/local/miniconda/bin/conda

...

  1. 
no change     /usr/local/miniconda/bin/conda-env

...

  1. 
no change     /usr/local/miniconda/bin/activate

...

  1. 
no change     /usr/local/miniconda/bin/deactivate

...

  1. 
no change     /usr/local/miniconda/etc/profile.d/conda.sh

...

  1. 
no change     /usr/local/miniconda/etc/fish/conf.d/conda.fish

...

  1. 
no change     /usr/local/miniconda/shell/condabin/Conda.psm1

...

  1. 
no change     /usr/local/miniconda/shell/condabin/conda-hook.ps1

...

  1. 
no change     /usr/local/miniconda/lib/python3.11/site-packages/xontrib/conda.xsh

...

  1. 
no change     /usr/local/miniconda/etc/profile.d/conda.csh

...

  1. 
modified      /home/USERNAME/.bashrc

...

  1. 

==> For changes to take effect, close and re-open your current shell. <==
  2. After restarting your shell, you should be able to see that you're in base environment of conda.
    Code Block
    languagebash
    titleActivate the conda environment

...

  1. and test the application
    (base) $ conda activate epct-desktop

(epct-desktop) $ epct info

...

  1. epct_version: 3.

...

  1. 4.0

...

  1. 
etc_dir: /usr/local/miniconda/envs/

...

  1. epct-desktop/etc/epct

...

  1. 
workspace_dir: /usr/local/miniconda/envs/

...

  1. epct-desktop/var/cache/epct

...

  1. 
customisations_log_dir: /usr/local/miniconda/envs/

...

  1. epct-desktop/var/log/epct

...

  1. 
output_dir: /

...

  1. 
installed_plugins:

...

  1. 
  - epct-plugin-netcdf-generator, 3.1.1
  - epct-plugin-ncarrays, 1.1.0

...

  1. 
  - epct-plugin-umarf, 3.2.0
  - epct-plugin-gis, 3.

...

  1. 3.0

...

  1. 
registered_backends:

...

  1. 
  - epct_1d_array

...

  1. 
  - epct_2d_array

...

  1. 
  - epct_ascatl1szf

...

  1. 
  - epct_eps_gome2l1b

...

  1. 
  - epct_eps_grasl1

...

  1. 
  - epct_eps_iasil1c

...

  1. 
  - epct_eps_iasisnd02

...

  1. 
  - epct_eps_native

...

  1. 
  - epct_eps_native_gome2l1_iasisnd02

...

  1. 
  - epct_gis

...

  1. 
  - epct_gis_eps

...

  1. 
  - epct_gis_eps_grib2

...

  1. 
  - epct_gis_fcil2
  - epct_gis_glbsst

...

  1. 
  - epct_gis_glbsst_grib2

...

  1. 
  - epct_gis_grib2

...

  1. 
  - epct_gis_hrit

...

  1. 
  - epct_gis_hrit_grib2

...

  1. 
  - epct_gis_hrit_png_16_bit
  - epct_gis_msg

...

  1. 
  - epct_gis_msg_meteo_bufr
  - epct_gis_msg_png_16_bit
  - epct_gis_mtgfcil1
  - epct_or1sww025
  - epct_plugin_netcdf_generator
  - epct_plugin_umarf_isccp
  - epct_plugin_umarf_isccp_nat
  - epct_plugin_umarf_msgclmk_grib
  - epct_plugin_umarf_msgclmk_netcdf
  - epct_plugin_umarf_msgnative
  - epct_plugin_umarf_msgnative_hrv_nat
  - epct_plugin_umarf_msgnative_nat
  - epct_plugin_umarf_openmtp
  - epct_plugin_umarf_rgb_dc_jpeg
  - epct_plugin_umarf_rgb_dc_jpeg_nat
  - epct_plugin_umarf_rgb_dc_png
  - epct_plugin_umarf_rgb_dc_png_nat
  - epct_plugin_umarf_xrit
  - epct_plugin_umarf_xrit_nat
executable_paths:
  epct_plugin_netcdf_generator: null

The Data Tailor Standalone is ready to use. Whenever you want to use the application, just activate your conda environment with $ conda activate epct-desktop and run your customisations.

Usage

The Data Tailor standalone has multiple ways of use including:

For more information, please refer to:  Data Tailor Standalone Guide.

Example

Below is a small example of using the application with CLI. 

  1. Activate the conda environment.
    Code Block
    languagebash
    titleActivate the conda environment
    (base) $ conda activate epct-desktop
  2. Create the chain.yaml
    Code Block
    languageyml
    titlechain.yaml
    product: HRSEVIRI
format: netcdf4
projection: geographic
roi: western_europe
filter:
  bands:
    - channel_9
    - channel_10
    - channel_11
  3. Run your customisation using the command below:
    Code Block
    languagebash
    titleRun chain
    (epct-desktop) $ epct run-chain -f chain.yaml MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA.zip
2024-06-13 10:05:25 - PROCESSING.chain_runner[291] - INFO - Start process "4c15df87"
2024-06-13 10:05:25 - PROCESSING.chain_runner[241] - INFO - WORKER: localhost
2024-06-13 10:05:25 - PROCESSING.chain_runner[242] - INFO - PID: 17361
2024-06-13 10:05:25 - PROCESSING.chain_runner[243] - INFO - backend: epct_gis_msg
2024-06-13 10:05:25 - PROCESSING.chain_runner[244] - INFO - user: None
2024-06-13 10:05:28 - PROCESSING.preprocessing[677] - INFO - Processing details - product: HRSEVIRI chain-name: None chain-details: -product: HRSEVIRI -format: NetCDF4 (simplified) -projection: Geographic / Plate-Carree -roi: Western Europe -filter: Custom
2024-06-13 10:05:28 - PROCESSING.preprocessing[683] - INFO - ROI name: Western Europe - NSWE: [67, 35, -23, 16] - process: 4c15df87
2024-06-13 10:05:28 - PROCESSING.preprocessing[684] - INFO - Input products: MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA.nat
2024-06-13 10:05:28 - PROCESSING.epct_gis[742] - INFO - Expected steps: 6
2024-06-13 10:05:28 - PROCESSING.epct_gis[744] - INFO - Starting step "IMPORT" 1/6 ...
2024-06-13 10:05:28 - PROCESSING.vrt[90] - INFO - Command line and its output ...

gdal_translate -of VRT -a_srs "+proj=geos +h=35785831 +lon_0=0.0 +ellps=GRS80 +no_defs" -a_ullr -5565747.7 5565747.7 5565747.7 -5565747.7 -b 9 -b 10 -b 11  RAD:/usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/decompressed_data/MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA.nat /usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA.vrt

2024-06-13 10:05:28 - PROCESSING.epct_gis[397] - INFO - a preliminary ROI is applied to the intermediate VRT
2024-06-13 10:05:28 - PROCESSING.vrt[90] - INFO - Command line and its output ...

gdalwarp -cutline /usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/roi.shp -crop_to_cutline -cblend 10 -r near --config OGR_ENABLE_PARTIAL_REPROJECTION YES /usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA.vrt /usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA_roi.vrt

2024-06-13 10:05:28 - PROCESSING.epct_gis[747] - INFO - ... step "IMPORT" finished!
2024-06-13 10:05:28 - PROCESSING.epct_gis[754] - INFO - Starting step "FILTER" 2/6 ...
2024-06-13 10:05:28 - PROCESSING.epct_gis[758] - INFO - ... step "FILTER" finished!
2024-06-13 10:05:28 - PROCESSING.epct_gis[754] - INFO - Starting step "PROJECTION" 3/6 ...
2024-06-13 10:05:28 - PROCESSING.vrt[90] - INFO - Command line and its output ...

gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -of GTiff -co BIGTIFF=IF_SAFER -co COMPRESS=DEFLATE -ot Float32 -t_srs "EPSG:4326" --config THRESHOLD 0.005 -r near -srcnodata -1000.0 "/usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA_roi.vrt" "/usr/local/miniconda/envs/epct-desktop/var/cache/epct

...

  1. /EPCT_HRSEVIRI_FPC_4c15df87/intermediate_warped.tif"

2024-06-13 10:05:32 - PROCESSING.epct_gis[647] - INFO - ... step "PROJECTION" finished!
2024-06-13 10:05:32 - PROCESSING.epct_gis[754] - INFO - Starting step "ROI" 4/6 ...
2024-06-13 10:05:32 - PROCESSING.vrt[90] - INFO - Command line and its output ...

gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -te -23.0 35.0 16.0 67.0 -te_srs "EPSG:4326" --config THRESHOLD 0.005 -r near "/usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/intermediate_warped.tif" "/usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/intermediate_warped_cut.tif"

2024-06-13 10:05:32 - PROCESSING.epct_gis[647] - INFO - ... step "ROI" finished!
2024-06-13 10:05:32 - PROCESSING.epct_gis[754] - INFO - Starting step "FORMAT" 5/6 ...
2024-06-13 10:05:32 - PROCESSING.vrt[90] - INFO - Command line and its output ...

gdal_translate -of netCDF -co FORMAT=NC4 -ot Float32 "/usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/intermediate_warped_cut.tif" "/usr/local/miniconda/envs/epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_4c15df87/OUTPUTS/EPCT_HRSEVIRI_FPC_4c15df87.nc"

2024-06-13 10:05:32 - PROCESSING.epct_gis[647] - INFO - ... step "FORMAT" finished!
2024-06-13 10:05:32 - PROCESSING.postprocessing[468] - INFO - Starting step "POST-PROCESSING" 6/6 ...
2024-06-13 10:05:32 - PROCESSING.postprocessing[493] - INFO - ... step "POST-PROCESSING" finished!
2024-06-13 10:05:32 - PROCESSING.postprocessing[495] - INFO - output-product: ./HRSEVIRI_20240613T094510Z_20240613T095742Z_epct_4c15df87_FPC.nc
2024-06-13 10:05:32 - PROCESSING.postprocessing[503] - INFO - customisation time: 7 - process: 4c15df87
2024-06-13 10:05:32 - PROCESSING.postprocessing[504] - INFO - *** STOP PROCESSING - Status DONE ***

processing-time: 6.8 [sec]
Info

The HRSEVIRI product (MSG3-SEVI-MSG15-0100-NA-20240613095742.896000000Z-NA.zip) is downloaded directly from the Data Store (using eumdac to be specific) and untouched. EUMDAC is included in the Data Tailor standalone instance. An example EUMDAC download command is:

$ eumdac download -c EO:EUM:DAT:MSG:HRSEVIRI --limit 1

Above command is downloading latest available product from HRSEVIRI collection.

...