Page History

...

In the EUMETSAT section of the European Weather Cloud, there's now an instance equipped with the Data Tailor Standalone and EUMETSAT Data Access Client (EUMDAC) together, which includes all the currently available plugins on a Rocky Linux system.  Below, you'll find a guide that covers the provisioning, a minimal post-installation process, and usage instructions.

The Data Tailor is a product customization toolbox designed to enable users to tailor satellite data to their specific needs. It offers the ability to subset and aggregate data products across space and time, filter layers, generate quick looks, reproject data onto new coordinate reference systems, and reformat data into widely used Geographic Information System (GIS) formats such as netCDF and GeoTIFF, as well as image formats like JPEG and PNG. The Data Tailor can currently customize data from various satellite collections, including METOP, MSG, and Sentinel-3. There are two versions of the Data Tailor: the Data Tailor Web Service (DTWS) and the Data Tailor Standalone. The DTWS allows users to customize products in the EUMETSAT Data Store before downloading them. On the other hand, the Data Tailor Standalone is an application that can be installed on your machine, enabling you to customize downloaded products. Unlike the Data Tailor Web Services, which can be used with EUMDAC or via https://tailor.eumetsat.int, the standalone version is generally faster and does not have limitations such as maximum concurrent jobs or workspace size. For more information on capabilities of the Data Tailor, please refer to to Data Tailor Standalone Guide on User Portal and for more information about the available products and customisations inside the Data Tailor, please go to Products and Customisations Available in the Data Tailor page.

...

Current limitations

Installation

Provisioning

Installation

Provisioning

1. Create the Data Tailor standalone instance by clicking on + ADD button on your Morpheus Dashboard.
2. Search for Data Tailor Standalone, select it and click next.
   Image Added
3. Choose a NAME for the instance type and move to next step.
   Image Added
4.  Insert at least the following
5. Create the Data Tailor standalone instance by clicking on + ADD button on your Morpheus Dashboard.
6. Search for Data Tailor Standalone, select it and click next.
   Image Removed
7. Choose a NAME for the instance type and move to next step.
   Image Removed
8.  Insert at least the following inputs and click next:
   • layout: make sure it is rocky-datatailor
   • plan: select one with at least 16GB of RAM (e.g. eo2.xlarge)
   • network: private
   • security group: ssh
     
9. Once done, click Complete and the provisioning will start.
10. Look for the instance list (Provisioning → Instances) to check the status of itthe instance.
11. Once ready you will see the green Running sign below the instance as below:
    

...

1. SSH into your instance following the Remote Access to the VM documentation. Alternatively, you can also use the console on Morpheus from your instance (Provisioning → Instances → select your instance and go to the Console section)
2. Running the following commands you can verify if the installation was successful.

   Code Block
   language bash title Check conda installation and dt_3_3 environment Check conda installation and epct-desktop environment
   $ conda env list # conda environments: # base                  *  /usr/local/miniconda epct-desktop             /usr/local/miniconda/envs/epct-desktop

3. After confirming the installation, add conda environment to your shell and then restart it.
   

   Code Block
   language bash title Integrate conda in your shell and restart your shell
   $ conda init no change     /usr/local/miniconda/condabin/conda no change     /usr/local/miniconda/bin/conda no change     /usr/local/miniconda/bin/conda-env no change     /usr/local/miniconda/bin/activate no change     /usr/local/miniconda/bin/deactivate no change     /usr/local/miniconda/etc/profile.d/conda.sh no change     /usr/local/miniconda/etc/fish/conf.d/conda.fish no change     /usr/local/miniconda/shell/condabin/Conda.psm1 no change     /usr/local/miniconda/shell/condabin/conda-hook.ps1 no change     /usr/local/miniconda/lib/python3.11/site-packages/xontrib/conda.xsh no change   $ conda env list # conda environments: # base                  *  /usr/local/miniconda dt_3_3                   /usr/local/miniconda/envs/dt_3_3

   After confirming the installation, add conda environment to your shell and then restart it

   /etc/profile.d/conda.csh modified      /home/USERNAME/.bashrc ==> For changes to take effect, close and re-open your current shell. <==

4. After restarting your shell, you should be able to see that you're in base environment of conda.
   

   Code Block
   language bash title Integrate conda in your shell and restart your shell Activate the conda environment and test the application
   (base) $ conda activate epct-desktop (epct-desktop) $ epct info epct_version: 3.4.0 etc_dir:$ conda init no change     /usr/local/miniconda/condabin/conda no change     /usr/local/miniconda/bin/conda no change     /usr/local/miniconda/binenvs/conda-env no change    epct-desktop/etc/epct workspace_dir: /usr/local/miniconda/bin/activate no change    /envs/epct-desktop/var/cache/epct customisations_log_dir: /usr/local/miniconda/bin/deactivate no change     /usr/local/miniconda/etc/profile.d/conda.sh no change     /usr/local/miniconda/etc/fish/conf.d/conda.fish no change     /usr/local/miniconda/shell/condabin/Conda.psm1 no change     /usr/local/miniconda/shell/condabin/conda-hook.ps1 no change     /usr/local/miniconda/lib/python3.11/site-packages/xontrib/conda.xsh no change     /usr/local/miniconda/etc/profile.d/conda.csh modified      /home/USERNAME/.bashrc ==> For changes to take effect, close and re-open your current shell. <==

   After restarting your shell, you should be able to see that you're in base environment of conda.
   

   Code Block
   language bash title Activate the conda environment and test the application
   (base) $ conda activate dt_3_3 (dt_3_3) $ epct info epct_version: 3.3.0 etc_dir: /usr/local/miniconda/envs/dt_3_3/etc/epct workspace_dir: /usr/local/miniconda/envs/dt_3_3/var/cache/epct customisations_log_dir: /usr/local/miniconda/envs/dt_3_3/var/log/epct output_dir: / installed_plugins:   - epct-plugin-ncarrays, 1.1.0   - epct-plugin-gis, 3.2.0 registered_backends:   - epct_1d_array   - epct_2d_array   - epct_ascatl1szf   - epct_eps_gome2l1b   - epct_eps_grasl1   - epct_eps_iasil1c   - epct_eps_iasisnd02   - epct_eps_native   - epct_eps_native_gome2l1_iasisnd02   - epct_gis   - epct_gis_eps   - epct_gis_eps_grib2   - epct_gis_glbsst   - epct_gis_glbsst_grib2   - epct_gis_grib2   - epct_gis_hrit   - epct_gis_hrit_grib2   - epct_gis_msg   - epct_gis_msg_meteo_bufr   - epct_or1sww025miniconda/envs/epct-desktop/var/log/epct output_dir: / installed_plugins: - epct-plugin-netcdf-generator, 3.1.1 - epct-plugin-ncarrays, 1.1.0 - epct-plugin-umarf, 3.2.0 - epct-plugin-gis, 3.3.0 registered_backends: - epct_1d_array - epct_2d_array - epct_ascatl1szf - epct_eps_gome2l1b - epct_eps_grasl1 - epct_eps_iasil1c - epct_eps_iasisnd02 - epct_eps_native - epct_eps_native_gome2l1_iasisnd02 - epct_gis - epct_gis_eps - epct_gis_eps_grib2 - epct_gis_fcil2 - epct_gis_glbsst - epct_gis_glbsst_grib2 - epct_gis_grib2 - epct_gis_hrit - epct_gis_hrit_grib2 - epct_gis_hrit_png_16_bit - epct_gis_msg - epct_gis_msg_meteo_bufr - epct_gis_msg_png_16_bit - epct_gis_mtgfcil1 - epct_or1sww025 - epct_plugin_netcdf_generator - epct_plugin_umarf_isccp - epct_plugin_umarf_isccp_nat - epct_plugin_umarf_msgclmk_grib - epct_plugin_umarf_msgclmk_netcdf - epct_plugin_umarf_msgnative - epct_plugin_umarf_msgnative_hrv_nat - epct_plugin_umarf_msgnative_nat - epct_plugin_umarf_openmtp - epct_plugin_umarf_rgb_dc_jpeg - epct_plugin_umarf_rgb_dc_jpeg_nat - epct_plugin_umarf_rgb_dc_png - epct_plugin_umarf_rgb_dc_png_nat - epct_plugin_umarf_xrit - epct_plugin_umarf_xrit_nat executable_paths: epct_plugin_netcdf_generator: null

The Data Tailor Standalone is ready to use. Whenever you want to use the application, just activate your conda environment with $ conda activate dt_3_3epct-desktop and run your customisations.

...

1. Activate the conda environment.
   

   Code Block
   language bash title Activate the conda environment
   (base) $ conda activate dt_3_3epct-desktop

2. Create the chain.yaml
   

   Code Block
   language yml title chain.yaml
   product: HRSEVIRI format: netcdf4 projection: geographic roi: western_europe filter: bands: - channel_9 - channel_10 - channel_11

3. Run your customization customisation using the command below:
   

   Code Block
   language bash title Run chain
   (dt_3_3epct-desktop) $ epct run-chain -f chain.yaml MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA.zip 2024-0306-0713 1110:3905:4625 - PROCESSING.chain_runner[286291] - INFO - Start process "39923ab44c15df87" 2024-0306-0713 1110:3905:4625 - PROCESSING.chain_runner[236241] - INFO - WORKER: localhost 2024-0306-0713 1110:3905:4625 - PROCESSING.chain_runner[237242] - INFO - PID: 18746417361 2024-0306-0713 1110:3905:4625 - PROCESSING.chain_runner[238243] - INFO - backend: epct_gis_msg 2024-0306-0713 1110:3905:4625 - PROCESSING.chain_runner[239244] - INFO - user: None 2024-0306-0713 1110:3905:4928 - PROCESSING.preprocessing[612677] - INFO - Processing details - product: HRSEVIRI chain-name: None chain-details: -product: HRSEVIRI -format: NetCDF4 (simplified) -projection: Geographic / Plate-Carree -roi: Western Europe -filter: Custom 2024-0306-0713 1110:3905:4928 - PROCESSING.preprocessing[618683] - INFO - ROI name: Western Europe - NSWE: [67, 35, -23, 16] - process: 39923ab44c15df87 2024-0306-0713 1110:3905:4928 - PROCESSING.preprocessing[619684] - INFO - Input products: MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA.nat 2024-0306-0713 1110:3905:4928 - PROCESSING.epct_gis[1452742] - INFO - Expected steps: 6 2024-0306-0713 1110:3905:4928 - PROCESSING.epct_gis[1454744] - INFO - Starting step "IMPORT" 1/6 ... 2024-0306-0713 1110:3905:4928 - PROCESSING.vrt[8990] - INFO - Command line and its output ... gdal_translate -of VRT -a_srs "+proj=geos +h=35785831 +lon_0=0.0 +ellps=GRS80 +no_defs" -a_ullr -5565747.7 5565747.7 5565747.7 -5565747.7 -b 9 -b 10 -b 11 RAD:/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/decompressed_data/MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA.nat /usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA.vrt 2024-0306-0713 1110:3905:4928 - PROCESSING.epct_gis[1091397] - INFO - a preliminary ROI is applied to the intermediate VRT 2024-0306-0713 1110:3905:4928 - PROCESSING.vrt[8990] - INFO - Command line and its output ... gdalwarp -cutline /usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/roi.shp -crop_to_cutline -cblend 10 -r bilinearnear --config OGR_ENABLE_PARTIAL_REPROJECTION YES /usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA.vrt /usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA_roi.vrt 2024-0306-0713 1110:3905:5028 - PROCESSING.epct_gis[1457747] - INFO - ... step "IMPORT" finished! 2024-0306-0713 1110:3905:5028 - PROCESSING.epct_gis[1464754] - INFO - Starting step "FILTER" 2/6 ... 2024-0306-0713 1110:3905:5028 - PROCESSING.epct_gis[1468758] - INFO - ... step "FILTER" finished! 2024-0306-0713 1110:3905:5028 - PROCESSING.epct_gis[1464754] - INFO - Starting step "INTERMEDIATE_REPROJECTIONPROJECTION" 3/6 ... 2024-0306-0713 1110:3905:5028 - PROCESSING.vrt[8990] - INFO - Command line and its output ... gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -of GTiff -co BIGTIFF=IF_SAFERIF_SAFER -co COMPRESS=DEFLATE -ot Float32 -t_srs "EPSG:4326" --config THRESHOLD 0.005 -r bilinear -co COMPRESS=DEFLATEnear -srcnodata -1000.0 "/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/MSG3-SEVI-MSG15-0100-NA-2024030711274220240613095742.143000000Z896000000Z-NA_roi.vrt" "/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/intermediate_warped.tif" 2024-0306-0713 1110:3905:5532 - PROCESSING.epct_gis[1134647] - INFO - ... step "INTERMEDIATE_REPROJECTIONPROJECTION" finished! 2024-0306-0713 1110:3905:5532 - PROCESSING.epct_gis[1464754] - INFO - Starting step "ROI" 4/6 ... 2024-0306-0713 1110:3905:5532 - PROCESSING.vrt[8990] - INFO - Command line and its output ... gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -te -23.0 35.0 16.0 67.0 -te_srs "EPSG:4326" --config THRESHOLD 0.005 -r bilinearnear "/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/intermediate_warped.tif" "/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/intermediate_warped_cut.tif" 2024-0306-0713 1110:3905:5532 - PROCESSING.epct_gis[1134647] - INFO - ... step "ROI" finished! 2024-0306-0713 1110:3905:5532 - PROCESSING.epct_gis[1464754] - INFO - Starting step "FORMAT" 5/6 ... 2024-0306-0713 1110:3905:5532 - PROCESSING.vrt[8990] - INFO - Command line and its output ... gdal_translate -ot Float32 -of netCDF -co FORMAT=NC4 -ot Float32 "/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/intermediate_warped_cut.tif" "/usr/local/miniconda/envs/dt_3_3epct-desktop/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab44c15df87/OUTPUTS/EPCT_HRSEVIRI_FPC_39923ab44c15df87.nc" 2024-0306-0713 1110:3905:5532 - PROCESSING.epct_gis[1134647] - INFO - ... step "FORMAT" finished! 2024-0306-0713 1110:3905:5532 - PROCESSING.postprocessing[468] - INFO - Starting step "POST-PROCESSING" 6/6 ... 2024-0306-0713 1110:3905:5532 - PROCESSING.postprocessing[493] - INFO - ... step "POST-PROCESSING" finished! 2024-0306-0713 1110:3905:5532 - PROCESSING.postprocessing[495] - INFO - output-product: ./HRSEVIRI_20240307T111509Z20240613T094510Z_20240307T112742Z20240613T095742Z_epct_39923ab44c15df87_FPC.nc 2024-0306-0713 1110:3905:5532 - PROCESSING.postprocessing[503] - INFO - customisation time: 97 - process: 39923ab44c15df87 2024-0306-0713 1110:3905:5532 - PROCESSING.postprocessing[504] - INFO - *** STOP PROCESSING - Status DONE *** processing-time: 96.48 [sec]