Message Box | ||||
---|---|---|---|---|
| ||||
ECMWF has implemented a new state-of-the-art data access infrastructure to host the Climate and Atmospheric Data Stores (CDS and ADS, respectively). All layers of the infrastructure are being modernised: the front-end web interface, the back-end software engine, and the underlying cloud infrastructure hosting the service and core data repositories. As part of this development, a new data store for the Copernicus Emergency Management Service (CEMS) has been created. The CEMS Early Warning Data Store (EWDS) will host all the historical and forecast information for floods and forest fires at European and Global levels. Users are encouraged to migrate their accounts and scripts to the new EWDS Beta before 26 September 2024, when the system will become operational. For more information, Please read: CEMS Early Warning Data Store (EWDS) coming soon! |
Message Box | ||||
---|---|---|---|---|
| ||||
Updates to the EWDS documentation are ongoing as the implementation takes place. Scripts and examples on this page are under review and may not be fully functional yet for the EWDS, so please be patient. Please send your feedback and/or report any issue/bug you may have encountered while using the new CDS/ADS/EWDS infrastructure by raising an enquiry ticket through our dedicated Support Portal (ECMWF login required) - make sure you select "Yes" to the question Is this request related to CDS-beta/ADS-beta/CEMS-EW-beta? on the Support Portal request form - this will help the Support team with triaging enquiries more efficiently. |
Table of Content Zone | |
---|---|
|
Set up the Python environment
If you have not done it yet, create a Python virtual environment.
Activate the conda environment and install the additional Phython package
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# activate the local environment. conda activate myenv # install the required packages conda install cartopy matplotlib # and the cdsapi pip install cdsapi |
Visualize EFAS products
Retrieve data
These exercises require EFAS model output parameters and auxiliary data. You can retrieve them using the EWDS API, as described in the code blocks below:
Model outputs
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import cdsapi c = cdsapi.Client() DATASET='efas-historical' request = { "hday": ["15","16","17","18"], "hmonth": "11", "hyear": "2020", "model_levels": "surface_level", "system_version": "version_5_0", "time": ["00:00","06:00","18:00"], "variable": "river_discharge_in_the_last_6_hours", "data_format": "netcdf", "download_format": "unarchived" } c.retrieve(DATASET,request).download('clim_2020111500.nc') |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import cdsapi c = cdsapi.Client() DATASET = 'efas-forecast' REQUEST = { 'originating_centre': 'ecmwf', 'product_type': 'ensemble_perturbed_forecasts', 'variable': 'river_discharge_in_the_last_6_hours', 'model_levels': 'surface_level', 'year': '2020', 'month': '11', 'day': '15', 'time': '00:00', 'leadtime_hour': [ '6', '12', '18', '24', '30', '36', '42', '48', '54', '60', '66', '72', ], "data_format": "netcdf", "download_format": "unarchived" } c.retrieve(DATASET,REQUEST).download('eue_2020111500.nc') |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import cdsapi c = cdsapi.Client() DATASET='efas-forecast' REQUEST={ 'originating_centre': 'ecmwf', 'product_type': 'high_resolution_forecast', 'variable': [ 'volumetric_soil_moisture' ], 'model_levels': 'soil_levels', 'year': '2019', 'month': '01', 'day': '30', 'time': '00:00', 'leadtime_hour': [ '6', '12', '18', '24', '30', '36', '42', '48', '54', '60', '66', '72', ], 'soil_level': [ '1', '2', '3', ], 'data_format':'netcdf', 'download_format':'unarchived' } c.retrieve(DATASET,REQUEST).download('eud_2019013000.nc') |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import cdsapi c = cdsapi.Client() # forecast snow depth water equivalent DATASET='efas-forecast' REQUEST={ 'format': 'netcdf', 'originating_centre': 'ecmwf', 'variable': 'snow_depth_water_equivalent', 'product_type': 'control_forecast', 'model_levels': 'surface_level', 'year': '2021', 'month': '01', 'day': '30', 'time': '00:00', 'leadtime_hour': [ '6', '12', '18', '24', '30', '36', '42', '48', '54', '60', '66', '72', '78', '84', '90', '96', '102', ], 'data_format':'netcdf', 'download_format':'unarchived' } c.retrieve(DATASET,REQUEST).download('esd_2021013000.nc') |
Auxiliary data
Info | |||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
One can download the auxiliary data by simplying requesting the data through a CDS API request. Note that area sub-selection is not currently possible for the auxiliary (or time-invariant) data. Therefore any 'area' included in the CDS API request will be ignored and the data downloaded will cover the full EFAS domain.
And then unzip the auxiliary.zip
|
Plot map discharge
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import matplotlib.pyplot as plt import cartopy.crs as ccrs import cartopy.feature as cf import xarray as xr var_name = "dis06" # Open the dataset (assuming it's already in WGS84) ds = xr.open_dataset('./clim_2020111500.nc') print(ds) cmap = plt.cm.get_cmap('jet').copy() cmap.set_under('white') # Using the default PlateCarree projection (WGS84) crs = ccrs.PlateCarree() # Set filter for visualizing only discharge above a threshold vmin = 20 # Select the variable at a specific time step ds = ds[var_name].isel(valid_time=1) # Plot map discharge > 20 m³/s fig, ax = plt.subplots(1, 1, subplot_kw={'projection': crs}, figsize=(20, 20)) ax.gridlines(crs=crs, linestyle="-") ax.coastlines() ax.add_feature(cf.BORDERS) sc = ds.plot(ax=ax, cmap=cmap, vmin=vmin, add_colorbar=False) ax.set_title(f'{ds.long_name} > {vmin} $m^3/s$') cbar = plt.colorbar(sc, shrink=.5) cbar.set_label(ds.GRIB_name) plt.show() |
Plot Soil Wetness Index Map
- The Soil Wetness Index is provided by Efas EFAS starting from 30/05/2024, so we donwon't need to calculate it after 30/05/2024 . Before that date. However, for data before that date, we have will need to calculate the Soil Wetness Index using the following script. It may take a huge Please note that this process may require a significant amount of memory and time ,.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import xarray as xr import matplotlib.pyplot as plt import cartopy.crs as ccrs import cartopy.feature as cf import numpy as np # Load the wilting point ,field capacity and soil depth datasets thmin1 = xr.open_dataset('./thmin_1_5.0.nc') thmin2 = xr.open_dataset('./thmin_2_5.0.nc') thmin3 = xr.open_dataset('./thmin_3_5.0.nc') thmax1 = xr.open_dataset('./thmax_1_5.0.nc') thmax2 = xr.open_dataset('./thmax_2_5.0.nc') thmax3 = xr.open_dataset('./thmax_3_5.0.nc') sd1 = xr.open_dataset('./sd_1_5.0.nc') sd2 = xr.open_dataset('./sd_2_5.0.nc') sd3 = xr.open_dataset('./sd_3_5.0.nc') # load the EFAS volumetric soil moisture dataset ds = xr.open_dataset('./eud_2019013000.nc') # Create a custom color map function def make_cmap(colors, position=None): import matplotlib as mpl if position is None: position = np.linspace(0, 1, len(colors)) cdict = {'red': [], 'green': [], 'blue': []} for pos, color in zip(position, colors): cdict['red'].append((pos, color[0]/255, color[0]/255)) cdict['green'].append((pos, color[1]/255, color[1]/255)) cdict['blue'].append((pos, color[2]/255, color[2]/255)) return mpl.colors.LinearSegmentedColormap('custom_cmap', cdict) # Calculate the mean volumetric soil moisture across all steps and layers mean_vsw = ds.vsw.mean(dim='step') th1 = mean_vsw.sel(soilLayer=1) th2 = mean_vsw.sel(soilLayer=2) # Retrieve soil depth values (assuming the values are stored correctly) sd1_vals = sd1.soil_depth_1 sd2_vals = sd2.soil_depth_2 # Compute wilting point and field capacity thmin = (thmin1.wilting_point_1 * sd1_vals + thmin2.wilting_point_2 * sd2_vals) / (sd1_vals + sd2_vals) thmax = (thmax1.field_capacity_1 * sd1_vals + thmax2.field_capacity_2 * sd2_vals) / (sd1_vals + sd2_vals) # Compute total soil moisture across layers smtot = (th1 * sd1_vals + th2 * sd2_vals) / (sd1_vals + sd2_vals) # Compute the Soil Wetness Index SM = ((smtot - thmin) / (thmax - thmin)) ds['SM'] = SM # Define the custom colormap colors = [(64, 0, 3), (133, 8, 3), (249, 0, 0), (250, 106, 0), (249, 210, 0), (189, 248, 255), (92, 138, 255), (0, 98, 255), (0, 0, 255), (0, 0, 88)] position = [0, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1] cmap = make_cmap(colors, position) # Use PlateCarree projection (WGS84) crs = ccrs.PlateCarree() # Plot the Soil Wetness Index fig, ax = plt.subplots(1, 1, subplot_kw={'projection': crs}, figsize=(15, 15)) ax.gridlines(crs=crs, linestyle="-") ax.coastlines() ax.add_feature(cf.BORDERS) # Plot the soil wetness index sc = ds['SM'].plot(ax=ax, cmap=cmap, add_colorbar=False) cbar = plt.colorbar(sc, shrink=0.5) cbar.set_label("Soil Wetness Index") plt.show() |
- Plot Soil Wetness Index By Efas
Code Block | ||||
---|---|---|---|---|
| ||||
import xarray as xr import matplotlib.pyplot as plt import cartopy.crs as ccrs import cartopy.feature as cf import numpy as np ds = xr.open_dataset('./Soil-Wetness-Index.nc') print(ds) def make_cmap(colors, position=None): import matplotlib as mpl if position is None: position = np.linspace(0, 1, len(colors)) cdict = {'red': [], 'green': [], 'blue': []} for pos, color in zip(position, colors): cdict['red'].append((pos, color[0]/255, color[0]/255)) cdict['green'].append((pos, color[1]/255, color[1]/255)) cdict['blue'].append((pos, color[2]/255, color[2]/255)) return mpl.colors.LinearSegmentedColormap('custom_cmap', cdict) # Define the custom colormap colors = [(64, 0, 3), (133, 8, 3), (249, 0, 0), (250, 106, 0), (249, 210, 0), (189, 248, 255), (92, 138, 255), (0, 98, 255), (0, 0, 255), (0, 0, 88)] position = [0, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1] cmap = make_cmap(colors, position) # Use PlateCarree projection (WGS84) crs = ccrs.PlateCarree() # Plot the Soil Wetness Index fig, ax = plt.subplots(1, 1, subplot_kw={'projection': crs}, figsize=(15, 15)) ax.gridlines(crs=crs, linestyle="-") ax.coastlines() ax.add_feature(cf.BORDERS) sc = ds['swir'].plot(ax=ax, cmap=cmap, add_colorbar=False) cbar = plt.colorbar(sc, shrink=0.5) cbar.set_label("Soil Wetness Index") plt.show() |
Plot Snow depth water equivalent map
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import xarray as xr ds = xr.open_dataset('./esd_2021013000.nc') ds.sd.plot(col="step",col_wrap=4, robust=True) |
Visualize GloFAS products
Also for this exercise, we are going to retrieve data from the CDS API, as described in the code blocks below:
Retrieve data
The script shows how to retrieve the control reforecasts product from 1999 2003 to 20182022, relative to the date 2019-01-03reference year 2023, for two station coordinates, one on the river network of the Thames and the other one on the Po river.
It's not possible to download only one cell of data , we have to provide and area .In the following script we are adding 0.05 degree to the station coordinates to get the surrounding area of the station . And for a correct interpretation of the data it's better to compare the uparea of the station with the uparea file provided .
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import cdsapi
from datetime import datetime, timedelta
def get_monthsdays():
start, end = datetime(2023, 3, 27), datetime(2023,12, 31) # reference year 2024
days = [start + timedelta(days=i) for i in range((end - start).days + 1)]
monthday = [d.strftime("%m-%d").split("-") for d in days if d.weekday() in [0, 3]]
return monthday
if __name__ == '__main__':
c = cdsapi.Client()
# set station coordinates (lat,lon)
COORDS = {
"Thames": [51.35, -0.45],
"Po": [44.85, 11.65]
}
adjustment = 0.05
bbox_dict = {}
for station, (lat, lon) in COORDS.items():
bbox = [
lat + adjustment, # North (Latitude + adjustment)
lon - adjustment, # West (Longitude - adjustment)
lat - adjustment, # South (Latitude - adjustment)
lon + adjustment # East (Longitude + adjustment)
]
bbox_dict[station] = bbox
DATASET='cems-glofas-reforecast'
# select date index corresponding to the event
MONTHSDAYS = get_monthsdays()
| ||||||
Code Block | ||||||
| ||||||
import cdsapi from datetime import datetime, timedelta def get_monthsdays(start =[2019,1,1],end=[2019,12,31]): # reforecast time index start, end = datetime(*start),datetime(*end) days = [start + timedelta(days=i) for i in range((end - start).days + 1)] monthday = [d.strftime("%m-%d").split("-") for d in days if d.weekday() in [0,3] ] return monthday # get date index MONTHSDAYS = get_monthsdays() if __name__ == '__main__': c = cdsapi.Client() # set station coordinates (lat,lon) COORDS = { "Thames":[51.35,-0.45], "Po":[44.85, 11.65] } # select all years YEARS = ['%d'%(y) for y in range(19992003,20192023)] # select all leadtime hours LEADTIMES = ['%d'%(l) for l in range(24,11281104,24)] # loop over date indexindex (just 1 in this case) for md in MONTHSDAYS: month = md[0].lower() day = md[1] # loop over station coordinates for station, bbox in COORDS: bbox_dict.items(): print(bbox) station_point_coord = COORDS[station]*2 # coordinates input for the area keyword REQUEST= { 'system_version': ["version_4_0"], c.retrieve( 'cems-glofas-reforecast', 'variable': 'river_discharge_in_the_last_24_hours', { 'systemhydrological_versionmodel': 'version_2_2lisflood', 'variableproduct_type': 'river_discharge_in_the_last_24_hourscontrol_reforecast', 'formatarea': 'grib', bbox, 'hydrological_modelhyear': 'htessel_lisflood'YEARS, 'product_typehmonth': 'control_reforecast'month, 'area':station_point_coord, 'hday': day, 'hyearleadtime_hour': YEARSLEADTIMES, 'hmonth 'data_format': month "grib2", 'hday': day , 'download_format': "zip" 'leadtime_hour': LEADTIMES, }, f'glofas_reforecastc.retrieve(DATASET, REQUEST).download(f'{DATASET}_{station}_{month}_{day}.gribzip') |
This second script shows how to retrieve a point time series reforecast on the river Thames for a single forecast reference time, specifically the 11th of July 2007.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
import cdsapi from datetime import datetime, timedelta if __name__ == '__main__': c = cdsapi.Client() COORDS = { "Thames": [51.35, -0.45], } adjustment = 0.05 bbox_dict = {} for station, (lat, lon) in COORDS.items(): bbox = [ lat + adjustment, # North (Latitude + adjustment) lon - adjustment, # West (Longitude - adjustment) lat - adjustment, # South (Latitude - adjustment) import cdsapi from datetime import datetime, timedelta def get_monthsdays(start =[2019,1,1],end=[2019,12,31]): # reforecast time index start, end = datetime(*start),datetime(*end) days = [start + timedelta(days=i) for i in range((end - start).days + 1)] monthday = [d.strftime("%m-%d").split("-") for d in days if d.weekday() in [0,3] ] return monthday if __name__ == '__main__': c = cdsapi.Client() # station coordinates (lat,lon) COORDS = { "Thames":[51.35,-0.45] lon } + adjustment # East (Longitude + #adjustment) select date index corresponding to the event ] MONTHSDAYS = get_monthsdays(start =[2019,7,11],end=[2019,7,11]) bbox_dict[station] = bbox YEAR = '2007' MONTH='03' DAY='27' LEADTIMES = ['%d'%(l) for l in range(24,1128,24)] # loop over station coordinates for station, bbox # loop over date index (just 1 in this case)in bbox_dict.items(): DATASET='cems-glofas-reforecast' for md in MONTHSDAYS: month = md[0].lower()REQUEST={ day = md[1] # loop over station coordinates 'system_version': ["version_4_0"], for station in COORDS: station_point_coord = COORDS[station]*2 # coordinates input for the area keyword 'variable': 'river_discharge_in_the_last_24_hours', c.retrieve('hydrological_model': 'lisflood', 'cems-glofas-reforecast', 'product_type': [ { 'system_version': 'version_2_2', "ensemble_perturbed_reforecast", 'variable': 'river_discharge_in_the_last_24_hours', "control_reforecast" 'format': 'grib' ], 'hydrological_modelarea': 'htessel_lisflood'bbox, 'product_typehyear': ['control_reforecast','ensemble_perturbed_reforecasts']YEAR, 'areahmonth':station_point_coord, MONTH, 'hyearhday': YEARDAY, 'hmonthleadtime_hour': month LEADTIMES, 'hdaydata_format': day "grib2", 'leadtimedownload_hourformat': LEADTIMES,"zip" }, f'glofas_reforecast_{station}_{month}_{day}.gribc.retrieve(DATASET, REQUEST).download(f'{DATASET}_{station}_{YEAR}_{MONTH}_{DAY}.zip') |
Plot ~20 years of reforecast time series from point coordinates
Code Block | ||
---|---|---|
| ||
import xarray as xr import numpy as np import matplotlib.pyplot as plt YEARS = range(19992004,20192024) # read dataset ds = xr.open_dataset("glofas_reforecast_Po_january_03.grib",engine="cfgrib") da = ds.iselsel(latitude=044.8962,longitude=011.64, method='nearest').dis24 # plotting parametersprint(da) n = len(YEARS) colors = plt.cm.jet(np.linspace(0,1,n)) ls = ["-","--",":"] # plot fig,ax = plt.subplots(1,1,figsize=(16,8)) steps = list(range(24,dsada.shape[1]*24+24,24)) for i,year in enumerate(YEARS): y = da.sel(time=f"{year}-01-0301").values ax.plot(steps,y,label=f"{year}-01-0301",color=colors[i],ls=ls[i%3]) plt.legend(bbox_to_anchor=(0.9,-0.1),ncol=7) ax.set_xlim([0,steps[-1]]) ax.set_xlabel("leadtime hours") ax.set_ylabel("m**3 s**-1") plt.title(f"Po river discharge at ({float(dsda.latitude.values),round(:.2f}, {float(dsda.longitude.values),2)}:.2f})") |
Plot reforecast ensemble members' time series for a historic flood event
In July 2007, a series of flooding events hit the UK, in particular in some areas of the upper Thames catchment. Up to 120 mm of rain fell between the 19th and 20th of July.
Code Block | ||||
---|---|---|---|---|
| ||||
import xarray as xr import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates from datetime import datetime,timedelta forecast_reference_time = datetime(2017,7,11) # read perturbed ensembles (dataType:pf) ds = xr.open_dataset("glofas_reforecast_Thames_july_11.grib",engine="cfgrib",backend_kwargs={'filter_by_keys':{'dataType': 'pf'}}) da = ds.isel(latitude=0,longitude=0).dis24 # read control (dataType:cf) cds = xr.open_dataset("glofas_reforecast_Thames_july_11.grib",engine="cfgrib",backend_kwargs={'filter_by_keys':{'dataType': 'cf'}}) cda = cds.isel(latitude=0,longitude=0).dis24 start = forecast_reference_time end = start + timedelta(45) time = pd.date_range(start,end) # plotting parameters ls = ["-","--",":"] locator = mdates.AutoDateLocator(minticks=2, maxticks=46) formatter = mdates.DateFormatter("%b-%d") # plot fig,ax = plt.subplots(1,1,figsize=(16,8)) ax.plot(time,cda.values,label=f"control",color="red") for i,number in enumerate(range(0,10)): y = da.isel(number=number).values ax.plot(time,y,label=f"ensemble {number}",color=colors[i],ls=ls[i%3]) plt.legend(bbox_to_anchor=(1,1),ncol=2) ax.axvspan(datetime(2017,7,19), datetime(2017,7,21), alpha=0.3, color='blue') # highlight event ax.set_xlabel("date") ax.set_ylabel("m**3 s**-1") ax.xaxis.set_major_locator(locator) ax.xaxis.set_major_formatter(formatter) ax.set_xlim([time[0],time[-1]]) plt.annotate("peak\nrainfall\nevent",(datetime(2017,7,19,3),85),rotation=0) plt.xticks(rotation=70) plt.title(f"Forecast reference time: 11-07-2007 - Thames river discharge at {float(ds.latitude.values),round(float(ds.longitude.values-360),2)}") |