...
- Install AVISO following instructions on: https://pyaviso.readthedocs.io/en/latest/guide/getting_started.html#installing
Configure MARS credentials following instructions on Access MARS - Create a configuration file in the default location ~/.avisomarsrc/config.yaml
Code Block username_file: mars.email and ~/.marsrc/mars.token files. mars.email file should contain “email” listed here: https://api.ecmwf.int/v1/key and mars.token file should contain “key” shown on the same page.language yml Create a configuration file in the default location ~/.aviso/config.yaml
Code Block language yml username_file: ~/.marsrc/mars.email key_file: ~/.marsrc/mars.token key_file: ~/.marsrc/mars.token notification_engine: type: etcd_rest host: aviso.ecmwf.int port: 443 https: true configuration_engine: type: etcd_rest host: aviso.ecmwf.int port: 443 https: true schema_parser: ecmwf remote_schema: True auth_type: ecmwf
- Create ~/.marsrc/mars.email and ~/.marsrc/mars.token files. mars.email file should contain “email” listed here: https://api.ecmwf.int/v1/key and mars.token file should contain “key” shown on the same page.
Create your your listener configuration file(s) with following content:- Example for dissemination:
Note theCode Block language yml title listener_diss.yaml listeners: - event: dissemination request: destination: <user_destination> class: od expver: 1 domain: g stream: enfo step: [1,2,3] triggers: - type: echo
dissemination
event listener.request
describes for which dissemination event users want to execute the triggers. It is made of a set of fields. Users have to specify only the fields they wants to use as filters.destination
is a mandatory field and it is associated to one or more destinations which are linked to the user's ECMWF account. Only the notifications complying with all the fields defined will execute the trigger. The trigger in this example isecho
. This will simply print out the notification to the console output. - Example for MARS:
Code Block language yml title listener_mars.yaml listeners: - event: mars request: class: od expver: 1 domain: g stream: enfo step: [1,2,3] triggers: - type: echo
Note, in the MARS event listener, destination field is not present.
You can see the the full list of allowed fields for MARS/ dissemination event listeners here.Warning Aviso will notify when data exists in MARS, but most users will not be able to retrieve this data before the scheduled time (Dissemination schedule). Triggering an automated retrieval of data will require adding logic to ensure you only retrieve data when you have respective permissions to do so. For automation of downstream time-critical workflows, this should mainly be based on dissemination events.
- Example for dissemination:
- If AVISO is installed in Python virtual environment, activate the environment first:
Code Block language bash source {PATH_TO_MY_ENV}/myenv/bin/activate
- Launch the AVISO application to listen ECMWF defined in your configuration file:
Code Block language bash aviso listen listener_diss.yaml #or aviso listen listener_mars.yaml
...
- The
dissemination
event is submitted by the product generation. The related listener configuration must define thedestination
field. A notification related to adissemination
event will have the fieldlocation
containinglocation
.By default, the location field contains the URL to the product notified . The
mars
event is designed for real-time data from the model output. The related listener configuration has no mandatory fields. Moreover the related notification will not contain thelocation
field because users will have to access to it by the conventional MARS API.
Request
The table below shows the full list of fields accepted in a request
block. These fields represent a subset on the MARS language.
- and can be used to download the file. However, it is also possible to configure your Dissemination Host in ECPDS to send full path to the local copy of transferred file. This can be done by adding following option to the Dissemination Host:
With this option, the location field will have value:Code Block language java ectrans.location = "file:"
Code Block language bash file:/{Full Path on the Destination File System}/{File Name}
The
mars
event is designed for real-time data from the model output. The related listener configuration has no mandatory fields. Moreover the related notification will not contain thelocation
field because users will have to access to it by the conventional MARS API.
Request
The table below shows the full list of fields accepted in a request
block. These fields represent a subset on the MARS language.
Field | Type | Event | Optional/Mandatory |
---|---|---|---|
destination | String, uppercase | dissemination | Mandatory |
target | String, uppercase | dissemination | |
Field | Type | Event | Optional/Mandatory |
destination | String, uppercase | dissemination | Mandatory |
target | String | dissemination | Optional |
class | Enum | All | Optional |
stream | Enum | All | Optional |
domain | Enum | All | Optional |
expver | Integer | All | Optional |
date | Date (e.g.20190810) | All | Optional |
time | Values: [0,6,12,18] | All | Optional |
step | Integer | All | Optional |
...
A notification is a dictionary whose keys can be used in the parameter substitution mechanism described above. Here an example of a notification:described above. Here an example of a notification:
Code Block | ||
---|---|---|
| ||
{
"event": "dissemination",
"request": {
"class": "od",
"date": "20191112",
"destination": "FOO",
"domain": "g",
"expver": "0001",
"step": "001",
"stream": "enfo",
"time": "18",
"target": "E1"
},
"location": "/home/ecpds/xxx.xx"
} |
Here an example file of a listener command triggering a bash script executing a MARS request.
Post
This trigger allows the user to send as HTTP POST message the notification received and formatted accordingly to the CloudEvents specification
Code Block | ||
---|---|---|
| ||
{triggers: - "event"type: "dissemination",post "request"protocol: { type: "class": "od", cloudevents_http "date": "20191112", "destination": "FOO", "domain": "g", url: http://my.notification.system/api |
This is the basic configuration. More parameters can be specified to customise the CloudEvents message. More info the reference documentation.
The CloudEvents message sent would look like the following:
Code Block | ||
---|---|---|
| ||
{ "type" : "aviso", "expver": "0001", "step": "001", "stream": "enfo", # this is customisable by "time": "18", "target": "E1" the user "data": { }, "location": "/home/ecpds/xxx.xx" } |
Here an example file of a listener command triggering a bash script executing a MARS request.
Post
This trigger allows the user to send as HTTP POST message the notification received and formatted accordingly to the CloudEvents specification
Code Block | ||
---|---|---|
| ||
triggers: - type: post protocol: type: cloudevents_http # url: http://my.notification.system/api |
This is the basic configuration. More parameters can be specified to customise the CloudEvents message. More info the reference documentation.
The CloudEvents message sent would look like the following:
Code Block | ||
---|---|---|
| ||
{ "type" : "avisothis is aviso specific "event": "dissemination", "request": { "target": "E1", # this is customisable by the user"class": "od", "data": { "date": "20190810", "destination": "FOO", "domain": "g", "expver": "1", "step": "1", # this is aviso specific "stream": "enfo", "eventtime": "dissemination0", }, "location": "/home/diss/foo/bar/20190810/xyz", # location on ceph or s3 "request": { }, "targetdatacontenttype": "E1application/json", "classid": "od0c02fdc5-148c-43b5-b2fa-cb1f590369ff", # UUID random generated by aviso "datesource": "20190810https://aviso.ecmwf.int", "destination": "FOO", # "domain": "g", this is customisable by the user "expverspecversion": "1.0", "steptime": "12020-03-02T13:34:40.245Z", "stream": "enfo", # Timestamp "time": "0", }, "location": "/home/diss/foo/bar/20190810/xyz", # location on ceph or s3 }, "datacontenttype": "application/json", "id": "0c02fdc5-148c-43b5-b2fa-cb1f590369ff", # UUID random generated by aviso "source": "https://aviso.ecmwf.int", # this is customisable by the user "specversion": "1.0", "time": "2020-03-02T13:34:40.245Z", # Timestamp of when this message is created } |
...
of when this message is created
} |
Aviso can also post events to a AWS Simple Notification Service (SNS) topic. See the relevant documentation page for more details: https://pyaviso.readthedocs.io/en/latest/reference/triggers.html#post
Combining multiple trigger types into sequence
An example of the triggers
block with multiple sequence of triggers is given below:
Code Block | ||
---|---|---|
| ||
triggers:
- type: echo
- type: log
path: testLog.log
- type: command
working_dir: $HOME/aviso/examples
command: ./script.sh --stream ${request.stream} --date ${request.date} --time ${request.time} --step ${request.step}
environment:
STEP: ${request.step}
TIME: "The time is ${request.time}" |
Aviso as a Python API
Aviso can be used as a Python API. This is intended for users that want to integrate Aviso in a bigger workflow written in Python or that simply have their trigger defined as a Python function. Below an example of a python script that defines a function to be executed once a notification is received, creates a listener that references to this function trigger and finally passes it to aviso to execute.
...