The ECPDS is the system used to produce and disseminate real-time forecast data to users - it is only available to Member & Co-operating States/Full NMHS/Maximum charge licence holders (under the old costing system)/ Silver or Gold Service Package licence holders (new Service Charge Model) and satellite data providers. 

What is ECPDS?

The ECMWF Production Data Store (ECPDS) is the system used to produce and disseminate real-time forecast data. Data retrieval and data distribution can be initiated by ECPDS using various protocols such as: ftp, sftp, ftps, http/s, Amazon S3, Microsoft Azure, dissftp and ECaccess/ECtrans. 

See an overview of the ECMWF Production Data Store (ECPDS).

This page explains how you can use the ECPDS to:

• Re-send files to your server or download them locally
• Push test files to your server or download them locally
• Check the status or progress of files transmission
• Modify priority within produced files
• View host configuration
• Get network information (trace route)
• View your organisation contact list

Access and login

Please login on https://diss-monitor.ecmwf.int/do/start

You will need fill in a username and password. For the password, you need to use a TOTP.  This is a six digit code which is generated via a smartphone. This is used instead of the password provided by ECMWF. You can configure the TOTP using the steps on this page: TOTP: How to activate. If you have any problems with access please contact us through the Support Portal.

In the past, we have used a security token. You can still login using this device, but we don't send out new tokens to users.

Training Material

A presentation and recording providing an overview of the ECPDS functionalities, along with guidance on how to begin using the system once access has been granted, are available as part of PREd & ECPDS Training Course Webinars: https://events.ecmwf.int/event/462/

These materials are intended to support users in understanding how to configure and maintain their real-time data requirements within PREd and review generated products via ECPDS.

The ECPDS interface

Click on Destinations and select the destination that you want to act on

Available information

• Status
  

  • Initialized: the Destination was just created and needs to be started

  • Retrying: the Destination is currently sending a file which failed before

  • Idle: there are no files to process in the queue for the time being

  • Stopped: the Destination was manually stopped

  • Running: the Destination is currently sending files

  • NoHosts: there is not a single Host defined or activated for this Destination

  • Waiting: there are files in the queue waiting to be processed

  • Restarting: the Destination was manually restarted

  • Sleeping: the Destination is sleeping before to enter the Retrying mode (when errors occurs the Destination can pause for some time before to retry)

• Last transfer
• Last Error
• Started
• Monitor: 'Yes' implies that our operators monitor this destination and will be aware of issues

Available filters

• Dissem_Str list all the "streams" or feed configured for this destination.
  
  • In parenthesis the number of files
  • On mouseover, you will see the volume
  • Click on one stream name will filter the display below and show only that stream
• Data_Str (file naming convention of MARS)
  
  • OPER
  • GENFO
  • .../...
• Status
  • Done (files successfully transferred)
  • Transmitting
  • StandBy (files not scheduled for automatic transfer. They can be pushed manually)
  • Queued (files scheduled for later transfer)
• Date
  • Default is today
  • all will show files for all dates
• Filename: free text to filter the display. Use regular expression syntax

Transfer table

The display shows the transfer table. The table can be ordered by clicking on any column:

• S (error status)
  
  • green tick: transfer ok
  • red cross: there was an error (the file may still have been transmitted)
• Host
• Sched. Time (Scheduled time) this is the earliest time at which the file can start being transmitted
• Start Time: this is the actual time at which a file started being transmitted
• Finish Time: Time at which the file transfer is completed
• Target: filename
• TS: Timestep
• %: show how much of the file has been transferred
• Bytes/s: the average speed at which the files were transferred
• Status: same as in the filter section
• Prior: priority. A low number indicates a high priority (files will be transferred before lower priority ones if scheduled at the same time). Priority is configured in the requirements and is configured per stream.
• Action:
  • Download locally
  • Re-queue: Send/Resent to host
  • Increase/decrease priority
• Select: allow to select individual files. The red arrow will display your current selection
• A/N/R at the bottom of the table
  • A select all files
  • N un-select all
  • R reverse selection

Easy operations you can do on your files

Stopping or restarting a destination

You would stop a destination if there is an outage period on your side. Stopping a destination makes that file transmission stops. After you restart it again, the files will be transferred automatically. Note that the lifetime of files on ECPDS is only 3 days. In case, your outage is for a longer period, you will need to extend the lifetime of the files (see 3. Extending the lifetime).

For stopping or restarting a destination, you use the following buttons, which you can find above the transfer table:

stops the destination gracefully (use this option if files are currently transferring, the destination will be stopped once all files have completed)

 - stops the destination immediately.

 - restarts the destination gracefully.

- restarts the destination immediately.

Push real-time data to yourself or stop files being pushed

Apply the filters to select only the data you want to transfer/stop transfer.

Files which have not been pushed before will have the status 'StandBy', files which are queued for pushing have the status 'Queued'. Files with the status 'done' have been sent.

In the example below, files are selected belonging to the 'TT' data feed from Thursday 19th March:

Scroll down to the bottom of the file list and select all files by pressing 'A' (see screenshot below)

Click on the Select all Pages (A) and go the the Selected Transfer Subset Icon (see image above). Please note, that it may take a while to load the next page.

Scroll down to the bottom of the page

• For pushing files, click on the green Requeue arrow - this triggers the transfer of the files.
• For stopping files click on Stop icon -this will unqueue the files from the transfer queue.

Extending the lifetime of a file

Select the files by applying the required filter and by clicking on A and on the icon, as explained under point 2) Push the data to yourself.
Scroll down to the bottom of the page and click on the Extend Transfer Lifetimeicon. This can take a long time to process, depending on how many files you are extending.

One click = one extra day.

You can press this button several times to extend by e.g. 3 days (be careful not to clog up ECPDS with 1000's of files).

Check if the lifetime is extended as required. You click on one of the files for which you have extended the lifetime. Under 'expiry date' it shows until which date the file will be there. 

Host configuration

This table lists the hosts associated with this destination. Note that we only send data to one host (the main host) based on priority (the lower the number the higher the priority). You can deactivate a host or change the priority. We need a backup host in case of failure to deliver to the main host. This will happen automatically.

To see the configuration details of each host click on the Host Name in the dissemination host(s) table:

If you want to send your data to your backup server (EC1-LAN-Host2 in our example)

• disable the main host (red cross)
• or change the priorities so that the back appears on the top.

The system will always try to send the data to the host with the highest priority (lowest number).

To activate a host which is currently disabled please select the green arrow to reactivate the host.

Get network information

Clicking on Network Info in the left-hand menu, will create a trace-route from our server to your server, which is useful for troubleshooting network issues.

Metadata

From the left-hand menu choose "xxx-Metadata" where xxx is the destination you are looking at. You will see your metadata, as shown below.

Please ensure that the contacts are up to date. If the information is not correct, contact the Data Support Team with new details.