EUMETSAT: Managing object storage (S3) bucket notifications

Object storage (S3) can be configured to send notifications when objects are created, modified or listed. The notifications can be sent to

1. HTTP endpoint
2. Kafka
3. AMQP

This documentation gives short summary of the functionality with minimalistic python examples. For the full functionality, the user is advised to consult Ceph documentation. While we recommend using python and boto3 library, notifications can be also configured using e.g. curl or AWS cli.    

In order to configure notifications, one must

1. Create topic to define the endpoint where notifications are sent
2. Set policy for the topic (optional, by default notifications are public)
3. Set up notifications for desired bucket and event

Create topic

Topic can be created following: 

#!/usr/bin/python3

import boto3

access_key_id = 'xx' # stored in Cypher
secret_access_key = 'xx' # stored in Cypher
ceph_endpoint = 'https://s3.waw3-1.cloudferro.com'
region_name = 'waw3-1' # required by boto3, any value works
endpoint = "http://vm-name.tenancy-name.s/f.ewcloud.host" # your own http endpoint
topic_name = 'example-topic'

s3 = boto3.client('sns',
                     region_name=region_name,
                     endpoint_url= ceph_endpoint,
                     aws_access_key_id=access_key_id,
                     aws_secret_access_key=secret_access_key)

s3.create_topic(Name=topic_name, Attributes={'persistent': 'True', 'push-endpoint': endpoint})

Push endpoint can get values in forms

• http[s]://<fqdn>[:<port]
• amqp[s]://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]
• kafka://[<user>:<password>@]<fqdn>[:<port]

See How to create S3 buckets in Morpheus - European Weather Cloud Knowledge Base - ECMWF Confluence Wiki for available ceph_endpoints.

Set topic policy

Set up notifications

List topics

Delete topics