Installation

Installation on EWC image Ubuntu 18.04.4 ECMWF built

Quick Start

Aviso can be used as a Python API or as Command-Line Interface (CLI) application. Here a few steps to quickly get a working configuration listening to notifications.

Note, this guide assumes the user to have a ECMWF account.

1. Make sure to have Python3 installed with pip3 support, setuptools and wheel.

Prepare the python3 environment needed

Code Block

language	bash
theme	DJango

user@local sudo apt update user@local sudo apt install build-essential user@local sudo apt-get install python3-dev user@local sudo apt-get install python3-pip user@local sudo pip3 install setuptools user@local sudo pip3 install wheel

3. Install Aviso

   Code Block
   language bash theme DJango
   user@local sudo pip3 install -e git+https://git.ecmwf.int/scm/lex/aviso.git@master#egg=aviso

Quick Start

...

1. 
   

2. Create a configuration file in the default location /etc/aviso/config.yaml with the following settings:

   Code Block
   language yml
   username: <user_email> listeners: - event: dissemination request: destination: <user_destination> stream: enfo step: [1,2,3] triggers: - type: echo

   The first line is required for a correct authentication, the username is the email associated to the user's ECMWF account. This can be checked by logging at https://api.ecmwf.int/v1/key/.

   This file is a basic example of a dissemination event listener. request describes for which dissemination events the user wants event users want to execute the triggers. It is made by a set of fields. Users have to specify only the fields that they wants to use as filters. destination is a mandatory field and it is associated to one or more destinations which are linked to the user's ECMWF account. Only the notifications complying with all the fields defined will execute the trigger. The trigger in this example is echo. This will simply print out the notification to the console output.

3. Save the ECMWF key as a file in /etc/aviso/key. The key can be obtained by logging at https://api.ecmwf.int/v1/key/ . 

4. Launch the aviso application

   Code Block
   language bash theme DJango
   user@local aviso listen

   Once in execution this command will create a process waiting for notifications. Users can terminate the application by typing CTRL+C .

   Note, the configuration file is only read -only at start time, therefore every time users make changes to it they need to restart the listening process.

...

Aviso provides the capability of submitting test notifications to a local server. This functionality can be used to test the listener configuration without any impact to the operational server.

1. Launch the aviso application in test mode. This allows to connect to a local file-based notification server, part of the aviso application, that is able to simulate the notification server behaviour.

   Code Block
   language bash theme DJango
   user@local aviso listen --test

   The console should display a Test Mode message.

2. Send a test dissemination notification. From another terminal run the notify command. Here an example, matching the example configuration presented above:

   Code Block
   language bash theme DJango
   user@local aviso notify event=dissemination,class=od,date=20190810,destination=<user_destination>,domain=g,expver=1,step=1,stream=enfo,time=0,location=xxxx --test

   Note the list of fields required for a dissemination event, the order is not important, but the command requires all of them. The destination has to match the one of the listener configuration. For

   
   

   Note, to submit a test mars event the  notification the fields destination and location have to be removed.

3. After a few seconds, the console output should display the notification, as the trigger is set to echo. 

Defining my listener

Aviso configuration file allows the definition of multiple listeners. Alternatively, the listeners configuration can be indicated as an independent file or multiple files:

...

• The dissemination event is submitted by the product generation. The related listener configuration must define the destination field. A notification received by a dissemination listener related to a dissemination event will have the field location containing the URL to the product notified.

• The mars event is designed for real-time data from the model output. The related listener configuration does not have the destination field and has no mandatory fields. Moreover the notification received by this listeners will related notification will not contain the location field because the users will be able have to access to it by the conventional MARS API.

...

The table below shows the full list of fields accepted in a request block. These fields represent a subset on the MARS language.

...

This is the simplest trigger as it prints the notification to the console output. It is used for testing and it does not accept any extra parameters.

...

This trigger logs the event to the a log file specified. It is useful for recording the received event. Note, it will fail if the directory does not exist.

...

This trigger allows the user to define a shell command to work with the notification.

triggers:
  - type: command
    working_dir: $HOME/aviso/examples
    command: ./script.sh --date ${request.date} -s ${request.stream}
    environment:
      STEP: ${request.step}
      TIME: "The time is ${request.time}"

...

Moreover, the system performs a parameter substitution in the command and environment fields, for every sequence of the patterns pattern:

•  ${name}

...

• , it replaces it with the value associated to the corresponding key found in the notification received.
• ${json}, it replaces it with the whole notification formatted as a JSON inline string.
• ${jsonpath}, it replaces it with the file name of a JSON file containing the notification.

A notification is a dictionary whose keys can be used in the parameter substitution mechanism described above. Here an example of a notification:

{
    "event": "dissemination",
    "request": {
        "class": "od",
        "date": "20191112",
        "destination": "FOO",
        "domain": "g",
        "expver": "0001",
        "step": "001",
        "stream": "enfo",
        "time": "18"
    },
    "location": "https://xxx.ecmwf.int/xxx/xxx.xx"
}    

The full notification can be passed in the command by using the keyword ${json} that will translate the structure in a JSON inline string. Finally, the trigger can save the notification to a JSON file whose file name can be retrieved using the keyword ${jsonpath}.

Aviso as a Python API

Aviso can be used as a Python API. This is intended for users that want to integrate Aviso in a bigger workflow written in Python or that simply have their trigger defined as a Python function. Below an example of a python script that defines a function to be executed once a notification is received, creates a listener that references to this function trigger and finally passes it to aviso to execute.The listening will happen in a background thread defined as daemon therefore it is responsibility of the user to keep the main thread alive.

Note, the Aviso configuration file /etc/aviso/config.yaml is still needed for the authentication as explained above.

from pyaviso import NotificationManager


# define function to be called
def do_something(notification):
    print(f"Notification for step {notification['request']['step']} received")
    # now do something useful with the notification
    it ...


# define the trigger
trigger = {"type": "function", "function": do_something}

# create a event listener request that uses that trigger
request = {"destination": "FOO", "stream": "enfo", "date": 20190810, "time": 0}
listener = {"event": "disseminationmars", "request": request, "triggers": [trigger]}
listeners = {"listeners": [listener]}

# run it
aviso = NotificationManager()
aviso.listen(listeners=listeners)

# wait ...

Dealing with historical  notifications

Before listening to new notifications, Aviso by default checks what was the last notification received and it will then return all the notifications that have been missed since. It will then carry on by listening to new ones. The first ever time the application runs however no previous notification will be returned. This behaviour allows users not to miss any notifications in case of machine reboots.

To override this behaviour by ignoring the missed notifications while listening only to the new ones, run the following:

user@local aviso listen --now

This command will also reset the previous history.

Users can also explicitly replay Aviso can also replay past notifications. The user Aviso can retrieve deliver notifications from the ECMWF server up to 14 days in the past. This can also be used to test the listener configuration with real notifications.​

...