Versions Compared

Old Version 15

changes.mady.by.user Yigit Oner Altintas

Saved on

compared with

New Version 16

changes.mady.by.user Francesco Murdaca

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents

Current limitations

Warning
  • The two of the Data Tailor plugins, epct_plugin_umarf and epct_plugin_netcdf_generator, are currently not available, hence they're not installed as default. We will put these plugins back in the installation as soon as they become available again.
  • Due to design of the software, only one linux user is able to use the software without additonal settings. When a user runs the application for the first time, various configuration files are created with permissions only editable by that user.


Installation

Provisioning

  1. Create the Data Tailor standalone instance by clicking on + ADD button on your Morpheus Dashboard.
  2. Search for Data Tailor Standalone, select it and click next.
  3. Confirm the layout. We recommend at least 16GB of ram in your instance for the application. Click next.
    Image Removed
  4. Create the instance and confirm it's running.
    Image Removed

Post-provisioning

SSH into your instance following the Remote Access to the VM documentation. You can also use the console on Morpheus.

Code Block
languagebash
titleCheck conda installation and dt_3_3 environment
$ conda env list
# conda environments:
#
base                  *  /usr/local/miniconda
dt_3_3                   /usr/local/miniconda/envs/dt_3_3

After confirming the installation, add conda environment to your shell.

...

languagebash
titleIntegrate conda in your shell and restart your shell
  1. Choose a NAME for the instance type and move to next step
    • Image Added
  2.  Insert at least the following inputs and click the next button:
    • layout: make sure is rocky-datatailor
    • plan: slect one with at least 16GB of RAM (e.g. eo2.xlarge)
    • network: private
    • security group: ssh
      Image Added
  3. Once done, click Complete and the provisioning will start.
  4. Look for the instance list (Provisioning → Instances) to check the status of it.
  5. Once ready you will see the green Running sign below the instance as below:
    Image Added

Post-provisioning




  1. SSH into your instance following the Remote Access to the VM documentation. Alternatively, you can also use the console on Morpheus from your instance (Provisioning → Instances → select your instance and go to the Console section)
  2. Running the following commands you can verify if the installation was successful.
    • Code Block
      languagebash
      titleCheck conda installation and dt_3_3 environment
      $ conda env list
# conda environments:
#
base                  *  /usr/local/miniconda
dt_3_3                  

...

    •  /usr/local/miniconda/

...

    • envs/dt_3_3
  2. After confirming the installation, add conda environment to your shell and then restart it.
    Code Block
    languagebash
    titleIntegrate conda in your shell and restart your shell
    $ conda init
no change     /usr/local/miniconda/

...

  1. condabin/

...

  1. conda
no change     /usr/local/miniconda/bin/

...

  1. conda
no change     /usr/local/miniconda/

...

  1. bin/conda

...

  1. -env
no change     /usr/local/miniconda/

...

  1. bin/activate
no change     /usr/local/miniconda/

...

  1. bin/deactivate
no change     /usr/local/miniconda/

...

  1. etc/

...

  1. profile.d/conda

...

  1. .

...

  1. sh
no change     /usr/local/miniconda/

...

  1. etc/fish/

...

  1. conf.

...

  1. d/conda.

...

  1. fish
no change     /usr/local/miniconda/

...

  1. shell/

...

  1. condabin/

...

  1. Conda.

...

  1. psm1

...

  1. no change    

...

  1. /usr/local/miniconda/shell/condabin/conda-hook.ps1
no change     /usr/local/miniconda/lib/python3.11/site-packages/xontrib/conda.xsh
no change     /usr/local/miniconda/etc/profile.d/conda.csh
modified      /home/USERNAME/.bashrc

==> For changes to take effect, close and re-open your current shell. <==
  2. After

...

  1. restarting your shell, you should be able to see that you're in base environment of conda.
Code Block
languagebash
titleActivate the conda environment and test the application
(base) $ conda activate dt_3_3

(dt_3_3) $ epct info
epct_version: 3.3.0
etc_dir: /usr/local/miniconda/envs/dt_3_3/etc/epct
workspace_dir: /usr/local/miniconda/envs/dt_3_3/var/cache/epct
customisations_log_dir: /usr/local/miniconda/envs/dt_3_3/var/log/epct
output_dir: /
installed_plugins:
  - epct-plugin-ncarrays, 1.1.0
  - epct-plugin-gis, 3.2.0
registered_backends:
  - epct_1d_array
  - epct_2d_array
  - epct_ascatl1szf
  - epct_eps_gome2l1b
  - epct_eps_grasl1
  - epct_eps_iasil1c
  - epct_eps_iasisnd02
  - epct_eps_native
  - epct_eps_native_gome2l1_iasisnd02
  - epct_gis
  - epct_gis_eps
  - epct_gis_eps_grib2
  - epct_gis_glbsst
  - epct_gis_glbsst_grib2
  - epct_gis_grib2
  - epct_gis_hrit
  - epct_gis_hrit_grib2
  - epct_gis_msg
  - epct_gis_msg_meteo_bufr
  - epct_or1sww025

The Data Tailor Standalone is ready to use. Whenever you want to use the application, just activate your conda environment with $ conda activate dt_3_3 and run your customisations.

...

The Data Tailor standalone has multiple ways of use including a web:

  •  web-app GUI (graphical user-interface)

...

  • CLI (command line interface), a Python Library and

...

For more information, please refer to:  Data Tailor Standalone Guide.

Example

Below is a small example of using the application with CLI. 

  1. Activate the conda environment
Code Block
languagebash
titleActivate the conda environment
(base) $ conda activate dt_3_3

Run your customisation.

) $ conda activate dt_3_3

2. Create the chain.yaml

Code Block
languageyml
titlechain.yaml
product: HRSEVIRI
format: netcdf4
projection: geographic
roi: western_europe
filter:
  bands:
    - channel_9
    - channel_10
    - channel_11

3. Run your customization using the command below:

Code Block
languagebash
titleRun chain
(dt_3_3) $ epct run-chain -f chain.yaml MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.zip
Code Block
languagebash
titleRun chain
(dt_3_3) $ epct run-chain -f chain.yaml MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.zip
2024-03-07 11:39:46 - PROCESSING.chain_runner[286] - INFO - Start process "39923ab4"
2024-03-07 11:39:46 - PROCESSING.chain_runner[236] - INFO - WORKER: localhost
2024-03-07 11:39:46 - PROCESSING.chain_runner[237] - INFO - PID: 187464
2024-03-07 11:39:46 - PROCESSING.chain_runner[238] - INFO - backend: epct_gis_msg
2024-03-07 11:39:46 - PROCESSING.chain_runner[239] - INFO - user: None
2024-03-07 11:39:49 - PROCESSING.preprocessing[612] - INFO - Processing details - product: HRSEVIRI chain-name: None chain-details: -product: HRSEVIRI -format: NetCDF4 (simplified) -projection: Geographic / Plate-Carree -roi: Western Europe -filter: Custom
2024-03-07 11:39:4946 - PROCESSING.preprocessingchain_runner[618286] - INFO - ROIStart name: Western Europe - NSWE: [67, 35, -23, 16] - process: 39923ab4process "39923ab4"
2024-03-07 11:39:46 - PROCESSING.chain_runner[236] - INFO - WORKER: localhost
2024-03-07 11:39:4946 - PROCESSING.preprocessingchain_runner[619237] - INFO - Input products: MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.natPID: 187464
2024-03-07 11:39:4946 - PROCESSING.epctchain_gisrunner[1452238] - INFO - Expected stepsbackend: 6epct_gis_msg
2024-03-07 11:39:4946 - PROCESSING.epctchain_gisrunner[1454239] - INFO - Starting step "IMPORT" 1/6 ... user: None
2024-03-07 11:39:49 - PROCESSING.vrtpreprocessing[89612] - INFO - Command line and its output ...
 
gdal_translate -of VRT -a_srs "+proj=geos +h=35785831 +lon_0=0.0 +ellps=GRS80 +no_defs" -a_ullr -5565747.7 5565747.7 5565747.7 -5565747.7 -b 9 -b 10 -b 11  RAD:/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/decompressed_data/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.nat /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/ Processing details - product: HRSEVIRI chain-name: None chain-details: -product: HRSEVIRI -format: NetCDF4 (simplified) -projection: Geographic / Plate-Carree -roi: Western Europe -filter: Custom
2024-03-07 11:39:49 - PROCESSING.preprocessing[618] - INFO - ROI name: Western Europe - NSWE: [67, 35, -23, 16] - process: 39923ab4
2024-03-07 11:39:49 - PROCESSING.preprocessing[619] - INFO - Input products: MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.vrtnat
 
2024-03-07 11:39:49 - PROCESSING.epct_gis[10911452] - INFO - Expected steps: 6
2024-03- a preliminary ROI is applied to the intermediate VRT07 11:39:49 - PROCESSING.epct_gis[1454] - INFO - Starting step "IMPORT" 1/6 ...
2024-03-07 11:39:49 - PROCESSING.vrt[89] - INFO - Command line and its output ...
 
gdalwarp -cutline /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/roi.shp -crop_to_cutline -cblend 10 -r bilinear --config OGR_ENABLE_PARTIAL_REPROJECTION YES /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.vrt gdal_translate -of VRT -a_srs "+proj=geos +h=35785831 +lon_0=0.0 +ellps=GRS80 +no_defs" -a_ullr -5565747.7 5565747.7 5565747.7 -5565747.7 -b 9 -b 10 -b 11  RAD:/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/decompressed_data/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA_roi.vrt
 
2024-03-07 11:39:50 - PROCESSING.epct_gis[1457] - INFO - ... step "IMPORT" finished!
2024-03-07 11:39:50 - PROCESSING.epct_gis[1464] - INFO - Starting step "FILTER" 2/6 ...
2024-03-07 11:39:50 - PROCESSING.epct_gis[1468] - INFO - ... step "FILTER" finished!.143000000Z-NA.nat /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.vrt
 
2024-03-07 11:39:5049 - PROCESSING.epct_gis[14641091] - INFO - Starting step "INTERMEDIATE_REPROJECTION" 3/6 ...a preliminary ROI is applied to the intermediate VRT
2024-03-07 11:39:5049 - PROCESSING.vrt[89] - INFO - Command line and its output ...
 
gdalwarp -overwrite -cutline /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/roi.shp -crop_to_cutline -cblend 10 -r bilinear --config CPLOGR_MAXENABLE_ERRORPARTIAL_REPORTSREPROJECTION 1 -of GTiff -co BIGTIFF=IF_SAFER -t_srs EPSG:4326 --config THRESHOLD 0.005 -r bilinear -co COMPRESS=DEFLATE -srcnodata -1000.0 "YES /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.vrt /usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA_roi.vrt" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped.tif"
 
-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA_roi.vrt
 
2024-03-07 11:39:50 - PROCESSING.epct_gis[1457] - INFO - ... step "IMPORT" finished!
2024-03-07 11:39:50 - PROCESSING.epct_gis[1464] - INFO - Starting step "FILTER" 2/6 ...
2024-03-07 11:39:5550 - PROCESSING.epct_gis[11341468] - INFO - ... step "INTERMEDIATE_REPROJECTIONFILTER" finished!
2024-03-07 11:39:5550 - PROCESSING.epct_gis[1464] - INFO - Starting step "ROIINTERMEDIATE_REPROJECTION" 43/6 ...
2024-03-07 11:39:5550 - PROCESSING.vrt[89] - INFO - Command line and its output ...
 
gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -teof GTiff -23.0 35.0 16.0 67.0co BIGTIFF=IF_SAFER -t_srs EPSG:4326 --config THRESHOLD 0.005 -r bilinear -co COMPRESS=DEFLATE -srcnodata -1000.0 "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped.tif39923ab4/MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA_roi.vrt" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped_cut.tif"
 
2024-03-07 11:39:55 - PROCESSING.epct_gis[1134] - INFO - ... step "ROIINTERMEDIATE_REPROJECTION" finished!
2024-03-07 11:39:55 - PROCESSING.epct_gis[1464] - INFO - Starting step "FORMATROI" 54/6 ...
2024-03-07 11:39:55 - PROCESSING.vrt[89] - INFO - Command line and its output ...
 
gdal_translate -ot Float32 -of netCDF -co FORMAT=NC4gdalwarp -overwrite --config CPL_MAX_ERROR_REPORTS 1 -te -23.0 35.0 16.0 67.0 --config THRESHOLD 0.005 -r bilinear "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped_cut.tif" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/OUTPUTS/EPCTintermediate_HRSEVIRI_FPC_39923ab4.nc"
 warped_cut.tif"
 
2024-03-07 11:39:55 - PROCESSING.epct_gis[1134] - INFO - ... step "ROI" finished!
2024-03-07 11:39:55 - PROCESSING.epct_gis[11341464] - INFO - Starting step "FORMAT" 5/6 ...
2024-03-07 11:39:55 - PROCESSING. step "FORMAT" finished!
2024-03-07 11:39:55 - PROCESSING.postprocessing[468] - INFO - Starting step "POST-PROCESSING" 6/6 ...vrt[89] - INFO - Command line and its output ...
 
gdal_translate -ot Float32 -of netCDF -co FORMAT=NC4 "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/intermediate_warped_cut.tif" "/usr/local/miniconda/envs/dt_3_3/var/cache/epct/EPCT_HRSEVIRI_FPC_39923ab4/OUTPUTS/EPCT_HRSEVIRI_FPC_39923ab4.nc"
 
2024-03-07 11:39:55 - PROCESSING.postprocessingepct_gis[4931134] - INFO - ... step "POST-PROCESSINGFORMAT" finished!
2024-03-07 11:39:55 - PROCESSING.postprocessing[495468] - INFO - output-product: ./HRSEVIRI_20240307T111509Z_20240307T112742Z_epct_39923ab4_FPC.nc Starting step "POST-PROCESSING" 6/6 ...
2024-03-07 11:39:55 - PROCESSING.postprocessing[503493] - INFO - customisation time: 9 - process: 39923ab4... step "POST-PROCESSING" finished!
2024-03-07 11:39:55 - PROCESSING.postprocessing[504495] - INFO - *** STOP PROCESSING - Status DONE ***
 
processing-time: 9.4 [sec]

The chain.yaml file is populated as follows:

Code Block
languageyml
titlechain.yaml
output-product: ./HRSEVIRI
format: netcdf4
projection: geographic
roi: western_europe
filter:
  bands:
    - channel_9
    - channel_10
    - channel_11
_20240307T111509Z_20240307T112742Z_epct_39923ab4_FPC.nc
2024-03-07 11:39:55 - PROCESSING.postprocessing[503] - INFO - customisation time: 9 - process: 39923ab4
2024-03-07 11:39:55 - PROCESSING.postprocessing[504] - INFO - *** STOP PROCESSING - Status DONE ***
 
processing-time: 9.4 [sec]
Info

The HRSEVIRI product (MSG3-SEVI-MSG15-0100-NA-20240307112742.143000000Z-NA.zip) is downloaded directly from the Data Store (using eumdac to be specific) and untouched.