Introduction
On the EUMETSAT side of European Weather Cloud, we have an instance that has the Data Tailor Standalone with all currently available plugins on a Rocky Linux system. Below is a small guide showing the provisioning, minimal post installation process and usage.
The Data Tailor is a product customization toolbox designed to allow users to modify satellite data according to their specific needs. It provides the capabilities to subset and aggregate data products in space and time, filter layers, generate quick looks, re-project onto new coordinate reference systems, and reformat data into common Geographic Information System (GIS) formats such as netCDF and GeoTIFF and image formats such as JPEG and PNG. The Data Tailor currently can customise various collections of METOP, MSG and Sentinel-3 satellites. There are two types of the Data Tailor. The Data Tailor Web Service (DTWS) allows users to perform customizations on products in the EUMETSAT Data Store before they are downloaded. The Data Tailor Standalone on the other hand is an application that you can install to your machine and customise products you downloaded. Comparing to Data Tailor Web Services that you can use with EUMDAC or through https://tailor.eumetsat.int, the Data Tailor standalone is generally faster and don't have the limitations like maximum concurrent jobs or workspace size. For more information on capabilities of the Data Tailor, please refer to Data Tailor Standalone Guide on User Portal and for more information about the available products and customisations inside the Data Tailor, please go to Products and Customisations Available in the Data Tailor page.
Current limitations
- The two of the Data Tailor plugins, epct_plugin_umarf and epct_plugin_netcdf_generator, are currently not available, hence they're not installed as default. We will put these plugins back in the installation as soon as they become available again.
- Due to design of the software, only one linux user is able to use the software without additonal settings. When a user runs the application for the first time, various configuration files are created with permissions only editable by that user.
Installation
Provisioning
- Create the Data Tailor standalone instance by clicking on + ADD button on your Morpheus Dashboard.
- Search for Data Tailor Standalone, select it and click next.
- Confirm the layout. We recommend at least 16GB of ram in your instance for the application. Click next.
- Create the instance and confirm it's running.
Post-provisioning
SSH into your instance following the Remote Access to the VM documentation. You can also use the console on Morpheus.
$ conda env list # conda environments: # base * /usr/local/miniconda dt_3_3 /usr/local/miniconda/envs/dt_3_3
After confirming the installation, add conda environment to your shell.
$ conda init no change /usr/local/miniconda/condabin/conda no change /usr/local/miniconda/bin/conda no change /usr/local/miniconda/bin/conda-env no change /usr/local/miniconda/bin/activate no change /usr/local/miniconda/bin/deactivate no change /usr/local/miniconda/etc/profile.d/conda.sh no change /usr/local/miniconda/etc/fish/conf.d/conda.fish no change /usr/local/miniconda/shell/condabin/Conda.psm1 no change /usr/local/miniconda/shell/condabin/conda-hook.ps1 no change /usr/local/miniconda/lib/python3.11/site-packages/xontrib/conda.xsh no change /usr/local/miniconda/etc/profile.d/conda.csh modified /home/USERNAME/.bashrc ==> For changes to take effect, close and re-open your current shell. <==
After restarting your shell, you should be able to see that you're in base environment of conda.
(base) $ conda activate dt_3_3 (dt_3_3) $ epct info epct_version: 3.3.0 etc_dir: /usr/local/miniconda/envs/dt_3_3/etc/epct workspace_dir: /usr/local/miniconda/envs/dt_3_3/var/cache/epct customisations_log_dir: /usr/local/miniconda/envs/dt_3_3/var/log/epct output_dir: / installed_plugins: - epct-plugin-ncarrays, 1.1.0 - epct-plugin-gis, 3.2.0 registered_backends: - epct_1d_array - epct_2d_array - epct_ascatl1szf - epct_eps_gome2l1b - epct_eps_grasl1 - epct_eps_iasil1c - epct_eps_iasisnd02 - epct_eps_native - epct_eps_native_gome2l1_iasisnd02 - epct_gis - epct_gis_eps - epct_gis_eps_grib2 - epct_gis_glbsst - epct_gis_glbsst_grib2 - epct_gis_grib2 - epct_gis_hrit - epct_gis_hrit_grib2 - epct_gis_msg - epct_gis_msg_meteo_bufr - epct_or1sww025
The Data Tailor Standalone is ready to use. Whenever you want to use the application, just activate your conda environment with $ conda activate dt_3_3
and run your customisations.
Usage
The Data Tailor standalone has multiple ways of use including a web-app GUI (graphical user-interface), CLI (command line interface), a Python Library and with EUMDAC. For more information, please refer to: Data Tailor Standalone Guide. Below is a small example of using the application with CLI.
(base) $ conda activate dt_3_3
Run your customisation.
(dt_3_3) $ epct run-chain -f chain.yaml MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.zip 2024-03-07 11:39:46 - PROCESSING.chain_runner[286] - INFO - Start process "39923ab4" 2024-03-07 11:39:46 - PROCESSING.chain_runner[236] - INFO - WORKER: localhost 2024-03-07 11:39:46 - PROCESSING.chain_runner[237] - INFO - PID: 187464 2024-03-07 11:39:46 - PROCESSING.chain_runner[238] - INFO - backend: epct_gis_msg 2024-03-07 11:39:46 - PROCESSING.chain_runner[239] - INFO - user: None 2024-03-07 11:39:49 - PROCESSING.preprocessing[612] - INFO - Processing details - product: HRSEVIRI chain-name: None chain-details: -product: HRSEVIRI -format: NetCDF4 (simplified) -projection: Geographic / Plate-Carree -roi: Western Europe -filter: Custom 2024-03-07 11:39:49 - PROCESSING.preprocessing[618] - INFO - ROI name: Western Europe - NSWE: [67, 35, -23, 16] - process: 39923ab4 2024-03-07 11:39:49 - PROCESSING.preprocessing[619] - INFO - Input products: MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.nat 2024-03-07 11:39:49 - PROCESSING.epct_gis[1452] - INFO - Expected steps: 6 2024-03-07 11:39:49 - PROCESSING.epct_gis[1454] - INFO - Starting step "IMPORT" 1/6 ... 2024-03-07 11:39:49 - PROCESSING.vrt[89] - INFO - Command line and its output ... gdal_translate -of VRT -a_srs "+proj=geos +h=35785831 +lon_0=0.0 +ellps=GRS80 +no_defs" -a_ullr -5565747.7 5565747.7 5565747.7 -5565747.7 -b 9 -b 10 -b 11 RAD:/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/decompressed_data/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.nat /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.vrt 2024-03-07 11:39:49 - PROCESSING.epct_gis[1091] - INFO - a preliminary ROI is applied to the intermediate VRT 2024-03-07 11:39:49 - PROCESSING.vrt[89] - INFO - Command line and its output ... gdalwarp -cutline /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/roi.shp -crop_to_cutline -cblend 10 -r bilinear --config OGR_ENABLE_PARTIAL_REPROJECTION YES /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.vrt /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA_roi.vrt 2024-03-07 11:39:50 - PROCESSING.epct_gis[1457] - INFO - ... step "IMPORT" finished! 2024-03-07 11:39:50 - PROCESSING.epct_gis[1464] - INFO - Starting step "FILTER" 2/6 ... 2024-03-07 11:39:50 - PROCESSING.epct_gis[1468] - INFO - ... step "FILTER" finished! 2024-03-07 11:39:50 - PROCESSING.epct_gis[1464] - INFO - Starting step "INTERMEDIATE_REPROJECTION" 3/6 ... 2024-03-07 11:39:50 - PROCESSING.vrt[89] - INFO - Command line and its output ... gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -of GTiff -co BIGTIFF=IF_SAFER -t_srs EPSG:4326 --config THRESHOLD 0.005 -r bilinear -co COMPRESS=DEFLATE -srcnodata -1000.0 "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA_roi.vrt" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped.tif" 2024-03-07 11:39:55 - PROCESSING.epct_gis[1134] - INFO - ... step "INTERMEDIATE_REPROJECTION" finished! 2024-03-07 11:39:55 - PROCESSING.epct_gis[1464] - INFO - Starting step "ROI" 4/6 ... 2024-03-07 11:39:55 - PROCESSING.vrt[89] - INFO - Command line and its output ... gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -te -23.0 35.0 16.0 67.0 --config THRESHOLD 0.005 -r bilinear "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped.tif" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped_cut.tif" 2024-03-07 11:39:55 - PROCESSING.epct_gis[1134] - INFO - ... step "ROI" finished! 2024-03-07 11:39:55 - PROCESSING.epct_gis[1464] - INFO - Starting step "FORMAT" 5/6 ... 2024-03-07 11:39:55 - PROCESSING.vrt[89] - INFO - Command line and its output ... gdal_translate -ot Float32 -of netCDF -co FORMAT=NC4 "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped_cut.tif" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/OUTPUTS/EPCT_HRSEVIRI_FPC_39923ab4.nc" 2024-03-07 11:39:55 - PROCESSING.epct_gis[1134] - INFO - ... step "FORMAT" finished! 2024-03-07 11:39:55 - PROCESSING.postprocessing[468] - INFO - Starting step "POST-PROCESSING" 6/6 ... 2024-03-07 11:39:55 - PROCESSING.postprocessing[493] - INFO - ... step "POST-PROCESSING" finished! 2024-03-07 11:39:55 - PROCESSING.postprocessing[495] - INFO - output-product: ./HRSEVIRI_20240307T111509Z_20240307T112742Z_epct_39923ab4_FPC.nc 2024-03-07 11:39:55 - PROCESSING.postprocessing[503] - INFO - customisation time: 9 - process: 39923ab4 2024-03-07 11:39:55 - PROCESSING.postprocessing[504] - INFO - *** STOP PROCESSING - Status DONE *** processing-time: 9.4 [sec]
The chain.yaml file is populated as follows:
(dt_3_3) $ vi chain.yaml product: HRSEVIRI format: netcdf4 projection: geographic roi: western_europe filter: bands: - channel_9 - channel_10 - channel_11
The HRSEVIRI product (MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.zip) is downloaded directly from the Data Store (using eumdac to be specific) and untouched.