Versions Compared

Old Version 9

changes.mady.by.user Roope Tervo

Saved on

compared with

New Version 10

changes.mady.by.user Roberto Cuccu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Panel
borderColorred
borderWidth1
borderStyledashed

This feature is in beta phase and may still experience issues

Panel
borderColorred
borderWidth1
borderStyledashed

This feature is currently available only at EUMETSAT side of the EWC

Table of Contents

Introduction

Object storage (S3) can be configured to send notifications when objects are created or deleted. The notifications can be sent to

...

  1. Create topic to define the endpoint where notifications are sent
  2. Set policy for the topic (optional, by default notifications are public)
  3. Set up notifications for desired bucket and event

Create topic

Topic can be created following: 

Code Block
languagepy
#!/usr/bin/python3

import boto3

access_key_id = 'xx' # stored in Cypher
secret_access_key = 'xx' # stored in Cypher
ceph_endpoint = 'https://s3.waw3-1.cloudferro.com<ceph endpoint>'
region_name = 'waw3-1default' # required by boto3, any value works
endpoint = "http://vm-name.tenancy-name.s/f.ewcloud.host" # your own http endpoint
topic_name = 'example-topic'

s3client = boto3.client('sns',
                     regionendpoint_nameurl=region ceph_nameendpoint,
                     endpoint_url= ceph_endpointaws_access_key_id=access_key_id,
                     aws_secret_access_key=secret_id=access_key_id,
                     awsregion_secret_access_key=secret_access_key)

s3name=region_name,
                     config=Config(signature_version='s3'))

client.create_topic(Name=topic_name, Attributes={'persistent': 'True', 'push-endpoint': endpoint})

The <ceph endpoint> shall be replaced with the proper one :

  1. The EUMETSAT endpoint is https://s3.waw3-1.cloudferro.com.
  2. The ECMWF endpoint are (depending on which cloud cluster is linked to the tenancy - this can be checked under Infrastructure->Clouds):


Push endpoint can get values in forms

...

Note that topic names are public

Set up notifications

Notifications can be configured following:

Code Block
languagepy
#!/usr/bin/python3
import boto3

access_key_id='xx' # from Cypher
secret_access_key='xx'  # from Cypher 
ceph_endpoint = "https://s3.waw3-1.cloudferro.com<ceph endpoint>"
region = 'waw3-1default'  # required by boto3, any value works 

bucket_name = "reflectivity.composite.opera.hdf5<bucket name>"

s3 = boto3.client('s3',
                endpoint_url=ceph_endpoint ,
                aws_access_key_id=access_key_id,
                aws_secret_access_key=secret_access_key)

# Create notification configuration
response = s3.put_bucket_notification_configuration(
    Bucket=bucket_name,
    NotificationConfiguration={
        'TopicConfigurations': [
            {
                'Id': 'test-http-topic',
                'TopicArn': 'arn:aws:sns:waw3-1default::example-topic',
                'Events': ['s3:ObjectCreated:*']
            }
        ]
    }
)

The <ceph endpoint> shall be replaced with the proper one :

  1. The EUMETSAT endpoint is https://s3.waw3-1.cloudferro.com.
  2. The ECMWF endpoint are (depending on which cloud cluster is linked to the tenancy - this can be checked under Infrastructure->Clouds):



TopicArn follows pattern arn:aws:sns:{region}::{topic-name}

...

Code Block
#!/usr/bin/python3
import boto3

access_key_id='xx' # from Cypher
secret_access_key='xx'  # from Cypher 
ceph_endpoint = "https://s3.waw3-1.cloudferro.com<ceph endpoint>"
region = 'waw3-1default'  # required by boto3, any value works 

bucket_name = "reflectivity.composite.opera.hdf5<bucket name>"

s3 = boto3.client('s3',
                endpoint_url=ceph_endpoint ,
                aws_access_key_id=access_key_id,
                aws_secret_access_key=secret_access_key)

# Create notification configuration
response = s3.put_bucket_notification_configuration(
    Bucket=bucket_name,
    NotificationConfiguration={
        'TopicConfigurations': [
            {
                'Id': 'test-http-topic',
                'TopicArn': 'arn:aws:sns:waw3-1default::example-topic',
                'Events': ['s3:ObjectCreated:*'],
                'Filter': {
                    'Key': {
                        'FilterRules': [
                            {
                                'Name': 'suffix',
                                'Value': 'reflectivity-composite-opera.h5'
                            }
                        ]
                    }
                }             
            }
        ]
    }
)

Note: the "Filter" section with the suffix value example " reflectivity-composite-opera.h5" is optional and could be excluded from the request payload.


See See Bucket Operations — Ceph Documentation for more detailed documentation.

List topics

Available topics can be listed following:

Code Block
languagepy
#!/usr/bin/python3

import boto3

access_key_id = 'xx' # stored in Cypher
secret_access_key = 'xx' # stored in Cypher

ceph_endpoint = 'https://s3.waw3-1.cloudferro.com<ceph endpoint>'
region_name = 'waw3-1default' # required by boto3, any value works

s3sns = boto3.client('sns',
  region_name=region_name,
  endpoint_url= ceph_endpoint,
  aws_access_key_id=access_key_id,
  aws_secret_access_key=secret_access_key,
  config=Config(signature_version='s3'))

response = s3sns.list_topics()

# Print the Topic ARNs
print('All topics:')
for topic in response['Topics']:
  print(' -'+topic['TopicArn'])

Delete topics

Topics can be deleted following:

Code Block
python
python
#!/usr/bin/python3
import boto3

access_key_id='xx' # from Cypher
secret_access_key='xx'  # from Cypher 
ceph_endpoint = "https://s3.waw3-1.cloudferro.com<ceph endpoint>"
region = 'waw3-1default'  # required by boto3, any value works 

arn = 'arn:aws:sns:waw3-1default::example-topic'

sns = boto3.client('sns',
                   region_name=region,
                   endpoint_url=ceph_endpoint ,
                   aws_access_key_id=access_key_id,
                   aws_secret_access_key=secret_access_key)

# Delete the SNS topic
response = sns.delete_topic(TopicArn=arn)

...