Data Dissemination

The installation and testing

Transition plan

of ECPDS, the ECMWF Production Data Store, in Bologna is now complete. ECPDS, which is described in a Newsletter article, takes care of both the dissemination and acquisition of data.

Migration Dates

The migration of

1. A gradual migration of individual data streams starting approximately 2 weeks before the switch-over of the operational forecast
2. A switch-over of all data streams at the same time as the switch-over of the operational forecast

We will confirm at a later date which option will be used.

To prepare for an operational ECPDS in Bologna, expected for October 2022, we are asking recipients of operational data to do the following.  

the dissemination service will take place gradually over a two-week period, starting on Tuesday 4th October and finishing on Tuesday 18th October. 

The following migration dates are available for eligible* users to choose:

• Tuesday 4th October
• Wednesday 5th October
• Thursday 6th October
• Tuesday 11th October ← If we do not hear from you, we will migrate your destination(s) on Tuesday 11th October
• Wednesday 12th October
• Thursday 13th October
• Tuesday 18th October ← this is the last possible migration date. Users with large files, and users receiving data via our proxy, will only be able to migrate on this date and will be contacted separately.

On the date selected, users will be migrated to ECPDS in Bologna from the 06 run on the specified day. If you do not normally receive products with the 06 run, the changes will take effect with the next successive run that you receive data for (12 or 18 on the same day, or 00 the next day). Once your dissemination has been migrated, your operational data will be disseminated from ECPDS in Bologna and you will no longer receive your operational data from ECPDS in Reading.

* Eligible users are those receiving ECMWF real-time data via dissemination. For users receiving other sources of data, the migration will take place on 18th October.

Choosing a migration date

Please note that the earlier you migrate, the more time we will have to support you in case of any unexpected issues. However, please also be aware of the risks associated with an earlier migration. Whilst we have endeavoured to stress test ECPDS in Bologna as much as possible by disseminating release candidate test data in a semi-operational capacity, we cannot guarantee the support level of external network links or the performance of the system at full capacity. In case of any unforeseen issues with these early migrations, we may revert back to disseminate products from ECPDS in Reading.

If you choose to wait until Tuesday 18th October, there will be no fall-back option and if you fail to receive products from ECPDS in Bologna on this day, you will not receive your products until the issues have been solved. Please also note that the Data Support team will have a limited capacity to help all users on this final migration day.

What do I need to do before the migration?

1. Please contact the Data Support Team via the Support Portal to inform them of your preferred migration date from the list of dates above.
2. We will conduct a final connectivity test to your operational hosts on Tuesday 27th September, transferring two files named "ecmwf-tst.bol" and "ecmwf-tst.md5" 
3. To facilitate a smooth migration, we encourage users to avoid making changes to their requirements between Tuesday 4th and Tuesday 18th October. Please note that a change freeze will be in place from Sunday 16th October.
4. In preparation for this migration, we have asked users to participate in a series of tests. If you have not completed the steps in the table below, please contact us immediately to arrange testing. 

Please note that the operational forecast will continue to originate from our Reading Data Centre until the operational switch on 18 October 2022. 

Indicative timeline 

The operational switch-over is currently expected for 18 October 2022. The migration of dissemination users to the ECPDS in Bologna is scheduled to happen in October, close to, or on the day of the operational forecast switch-over. We will update this page with full details of the migration plan and dates shortly.

What do I need to do on migration day?

The migration of dissemination users to ECPDS in Bologna will start on Tuesday 4th October 2022, from the 06 run over a two-week period. Please ensure that you have followed our setup instructions below before this date. You will no longer be able to receive your operational data from Reading after Tuesday 18th October. 

Guidelines for users receiving ECMWF Real-time Forecast Data

 Note: For users receiving any other sources of data (not included in the following list), the migration will take place on 18th October. If you are not sure if this applies to you, please contact our Data Support Team through the Support Portal for assistance.

Internet Dissemination

Please ensure that your firewall has been updated to accept connections from the following IPs of our Bologna Data Centre. You may need to consult with your organisation’s network or firewall administrators.

Internet:

• • 136.156.192.0/26
  • 136.156.193.0/26

For ftp we support the following ports:
21 (standard)
2121 (non standard)

For sftp we support:
22 (standard)
2222 (non standard)

Please prepare these changes to all of your internet hosts, including any backup hosts 

RMDCN Dissemination

Please ensure that your firewall has been updated to accept connections from the following IPs of our Bologna Data Centre. You may need to consult with your organisation’s network or firewall administrators.

RMDCN:

• • 136.156.196.0/26
  • 136.156.197.0/26

For ftp we support the following ports:
21 (standard)
2121 (non standard)

For sftp we support:
22 (standard)
2222 (non standard)

Please prepare these changes to all of your RMDCN hosts, including any backup hosts

Pull (Data Retrieval)

On your selected migration day, from the 06 run, you will need to retrieve your data from diss.ecmwf.int. Your data will no longer be available from dissemination.ecmwf.int.

Before the migration, please try connecting to the host diss.ecmwf.int with the same credentials that you use to access ECPDS in Reading:

• • • https://diss.ecmwf.int
    • sftp <userid>@diss.ecmwf.int 
    • ftp diss.ecmwf.int 

Other Data Pull Datasets

WMO essential (migrating on 11th October 06 run)

• • • Users accessing WMO Essential data will need to access data from https://diss.ecmwf.int/ from 11th October 06 run
    • Users can try accessing this link now and should find two test files, ecmwf-tst.bol and ecmwf-tst.md5

WMO additional (migrating on 11th October 06 run)

• • • Users accessing WMO Additional data will need to access data from https://diss.ecmwf.int/ from 11th October 06 run
    • Users can try accessing this link now and should find two test files, ecmwf-tst.bol and ecmwf-tst.md5

ACMAD (migrating on 11th October 06 run)

• • • Users accessing ACMAD datasets will need to access data from https://data.ecmwf.int from 18th October 06 run

Open Data (migrating on 18th October 06 run)

• • • Users accessing ECMWF Open Data will need to access data from https://data.ecmwf.int/forecasts from 18th October 06 run

CAMS

• • Please refer to the instructions at Accessing CAMS 47r3 technical test data (ATOS)

Proxy

All users receiving data via the proxy in Washington DC will migrate on Tuesday 18th October. 

EWCloud

No action is required. The European Weather Cloud has not yet been migrated to Bologna. Products will continue to be uploaded to the European Weather Cloud in Reading. Please continue to access your products as normal.

LAN

On your selected migration day*, your data will be available on our ATOS supercomputer in Bologna. User Services are setting up all LAN dissemination to the ATOS and will arrange testing with you. Please contact us via the Support Portal if you have not heard from us.

Your data will no longer be available from the CRAY supercomputer in Reading after you migrate to Bologna.

*Or on 18th October for destinations with large files.

ECaccess

Users have two options

1. 1. 1. ECMWF have now made available a new release of the ECaccess gateway for users. We encourage users to prepare these changes for the migration. Full details are provided on the following page: Releases - Gateway package.
      2. Alternatively, on your selected migration day, you will need to point your existing ECaccess gateway to retrieve data from Bologna. Please contact User Services via the Support Portal (software and computing) for instruction if you choose to do this.

User Services are contacting relevant users with instructions for ECaccess. Please contact us via the Support Portal if you have not heard from us

Action required by dissemination users/sites

Dissemination of test data in September 2022

Following the announcement of the release candidate test data, we are offering users scheduled test dates to receive test products from the Bologna Data Centre. 

Who is this test for?

This test is available to users who:

• • Receive data via internet or RMDCN dissemination AND
  • Receive any of the products listed on the Release Candidate Phase page

If you are unsure about your eligibility, please contact our Data Support Team through the Support Portal. Please note that ecaccess installation is still in progress and is not yet ready for testing.

The release candidate test data is now available  and can be accessed at any time. The purpose of these important scheduled tests is to simulate operations and network performance. As this test is only meaningful if as many users as possible participate, we strongly encourage all users to take part.  However, we cannot guarantee the consistency of transfer rates. 

If you are unable to participate in these tests, please contact our Data Support Team through the Support Portal to arrange an alternative date.

How the tests will work

For all interested users, we will push test data to your hosts from the ECMWF Product Data Store (ECPDS) in Bologna, Italy.

• • Test data will be produced from your operational requirements. 
  • Test data will be uploaded to your operational hosts. We can transfer the test data to different hosts and directories upon request.
  • To avoid overwriting your operational data, the filenames will end with '77'.
  • Data will be transferred close to the normal operational schedule, though we can delay the dissemination of these products upon request. However, please note that the release candidate data is not yet operational and we cannot guarantee the availability of these products.

Please note, this is only test data and should not be used for operational purposes. You will continue to receive your operational data from Reading, UK during these tests. After the operational switch-over on 18 October your operational data will originate from the Bologna data centre.

Please refer to the table below for the dates/runs that we are offering to disseminate automatically to users. If you would like test data to be sent to your host for the entire week, please let us know.

00  (including SEAS &* ENS-extended)

00  (including ENS-extended)

*Seasonal data is not yet available to test. Please refer to the Release Candidate Phase page to be notified when this data becomes available for testing.

What do I need to do before the test?

Please contact the Data Support Team via the support portal at least 5 working days before the test date to confirm:

1. 1. If you would like to be included in some or all of these tests
   2. If you would like to receive the test files to a different host and/or directory 
   3. If you would like to receive the test files with a delay, so that they do not interfere with your operational delivery

What do I need to do on test day?

For users with access to the ECMWF Product Data Store (ECPDS), you can track the progress of transfers through the following test interface in Bologna https://xdiss-monitor.ecmwf.int/do/start. If you require any assistance, please contact our Data Support Team through the Support Portal.

What should I do in case of problems?

Please be aware that you may notice different transfer rates compared with your usual rates from Reading.

We encourage all users to test the dissemination of real-time data from ECPDS in Bologna prior to migration by testing the release candidate phase test data.

If you experience any issues during your testing or during the migration, please contact our Data Support Team through our Support Portal for assistance.

If you require immediate assistance outside of UK working hours, please contact ECMWF Operators at operators@ecmwf.int for assistance.

We encourage users to follow the ECMWF Service Status page, where all updates will be posted throughout the migration period. We do not anticipate any issues with

the migration, but we will keep users informed and reserve the right to

amend the migration dates if needed.

Thank you for your ongoing participation and preparations for our migration to Bologna. To stay informed of any further updates

, please watch this page

Setup Instructions

Dissemination via the Internet or RMDCN

Please update your firewall rules ASAP to accept connections from our new IP addresses used in Bologna, Italy. You may need to consult with your organisation’s network or firewall administrators. Please note that if no action is taken, you risk not receiving ECMWF real-time data when we migrate operations to Bologna, Italy (currently scheduled for October 2022).

Internet:

• • 136.156.192.0/26
  • 136.156.193.0/26

RMDCN:

• • 136.156.196.0/26
  • 136.156.197.0/26

Please note that for ftp we support the following ports:
21 (standard)
2121 (non standard)

For sftp we support:
22 (standard)
2222 (non standard)

Please prepare these changes to your hosts, including any backup hosts, and ensure to keep the current firewall rules in place to allow connections from our Reading Data Centre subnets until its decommissioning in the third quarter of 2022.

Once you have prepared these changes, please contact the Data Support Team via our Support Portal so that we can arrange to test connectivity to your hosts.

Dissemination via ecaccess or LAN

Our installations are not yet complete in Bologna. We will update this page shortly to inform users when these services are available for testing. Please 'watch' this page to be notified of updates.

Dissemination via data pull (you download the data yourself)

Please try connecting to the host diss.ecmwf.int with the same credentials that you use to access ECPDS in Reading:

• • https://diss.ecmwf.int
  • sftp <userid>@diss.ecmf.int
  • ftp diss.ecmwf.int

.