This feature is currently available only at EUMETSAT side of the EWC

Introduction

Object storage (S3) can be configured to send notifications when objects are created or deleted. The notifications can be sent to

  1. HTTP endpoint
  2. Kafka
  3. AMQP

This documentation gives short summary of the functionality with minimalistic python examples. For the full functionality, the user is advised to consult Ceph documentation. While we recommend using python and boto3 library, notifications can be also configured using e.g. curl or AWS cli.    

In order to configure notifications, one must

  1. Create topic to define the endpoint where notifications are sent
  2. Set policy for the topic (optional, by default notifications are public)
  3. Set up notifications for desired bucket and event

Create topic

Topic can be created following: 

#!/usr/bin/python3

import boto3

access_key_id = 'xx' # stored in Cypher
secret_access_key = 'xx' # stored in Cypher
ceph_endpoint = 'https://s3.waw3-1.cloudferro.com'
region_name = 'waw3-1' # required by boto3, any value works
endpoint = "http://vm-name.tenancy-name.s/f.ewcloud.host" # your own http endpoint
topic_name = 'example-topic'

s3 = boto3.client('sns',
                     region_name=region_name,
                     endpoint_url= ceph_endpoint,
                     aws_access_key_id=access_key_id,
                     aws_secret_access_key=secret_access_key)

s3.create_topic(Name=topic_name, Attributes={'persistent': 'True', 'push-endpoint': endpoint})

Push endpoint can get values in forms

See How to create S3 buckets in Morpheus - European Weather Cloud Knowledge Base - ECMWF Confluence Wiki for available ceph_endpoints.

Note that topic names are public (TODO: what else is public?)

Set up notifications

Notifications can be configured following:

#!/usr/bin/python3
import boto3

access_key_id='xx' # from Cypher
secret_access_key='xx'  # from Cypher 
ceph_endpoint = "https://s3.waw3-1.cloudferro.com"
region = 'waw3-1'  # required by boto3, any value works 

bucket_name = "reflectivity.composite.opera.hdf5"

s3 = boto3.client('s3',
                endpoint_url=ceph_endpoint ,
                aws_access_key_id=access_key_id,
                aws_secret_access_key=secret_access_key)

# Create notification configuration
response = s3.put_bucket_notification_configuration(
    Bucket=bucket_name,
    NotificationConfiguration={
        'TopicConfigurations': [
            {
                'Id': 'test-http-topic',
                'TopicArn': 'arn:aws:sns:waw3-1::example-topic',
                'Events': ['s3:ObjectCreated:*']
            }
        ]
    }
)

TopicArn follows pattern arn:aws:sns:{region}::{topic-name}

Most typical event types are:

Please consult S3 Bucket Notifications Compatibility — Ceph Documentation for details.

Note, that one can also add filters for objects, e.g.: 

#!/usr/bin/python3
import boto3

access_key_id='xx' # from Cypher
secret_access_key='xx'  # from Cypher 
ceph_endpoint = "https://s3.waw3-1.cloudferro.com"
region = 'waw3-1'  # required by boto3, any value works 

bucket_name = "reflectivity.composite.opera.hdf5"

s3 = boto3.client('s3',
                endpoint_url=ceph_endpoint ,
                aws_access_key_id=access_key_id,
                aws_secret_access_key=secret_access_key)

# Create notification configuration
response = s3.put_bucket_notification_configuration(
    Bucket=bucket_name,
    NotificationConfiguration={
        'TopicConfigurations': [
            {
                'Id': 'test-http-topic',
                'TopicArn': 'arn:aws:sns:waw3-1::example-topic',
                'Events': ['s3:ObjectCreated:*'],
                'Filter': {
                    'Key': {
                        'FilterRules': [
                            {
                                'Name': 'suffix',
                                'Value': 'reflectivity-composite-opera.h5'
                            }
                        ]
                    }
                }             
            }
        ]
    }
)

See Bucket Operations — Ceph Documentation for more detailed documentation.

List topics

Available topics can be listed following:

#!/usr/bin/python3

import boto3

access_key_id = 'xx' # stored in Cypher
secret_access_key = 'xx' # stored in Cypher

ceph_endpoint = 'https://s3.waw3-1.cloudferro.com'
region_name = 'waw3-1' # required by boto3, any value works

s3 = boto3.client('sns',
region_name=region_name,
endpoint_url= ceph_endpoint,
aws_access_key_id=access_key_id,
aws_secret_access_key=secret_access_key)

response = s3.list_topics()

# Print the Topic ARNs
print('All topics:')
for topic in response['Topics']:
print(' -'+topic['TopicArn'])

Delete topics

Topics can be deleted following:

#!/usr/bin/python3
import boto3

access_key_id='xx' # from Cypher
secret_access_key='xx'  # from Cypher 
ceph_endpoint = "https://s3.waw3-1.cloudferro.com"
region = 'waw3-1'  # required by boto3, any value works 

arn = 'arn:aws:sns:waw3-1::example-topic'

sns = boto3.client('sns',
                   region_name=region,
                   endpoint_url=ceph_endpoint ,
                   aws_access_key_id=access_key_id,
                   aws_secret_access_key=secret_access_key)

# Delete the SNS topic
response = sns.delete_topic(TopicArn=arn)