Page History

Horizontal Navigation Bar

Button Group

Button Hyperlink

title	Previous
type	standard
url	https://confluence.ecmwf.int

HTML

<div class="section" id="using-python-scripting"> As you have already seen, ecFlow has a <a class="reference internal" href="/wiki

/display/ECFLOW/

ecFlow

Dates+

Python+Api#python-api">ecFlow Python Api</a>: <div class="highlight-python"><div class="highlight"><pre>#!/usr/bin/env python2.7 import ecflow </pre></div> </div> <div class="line-block"> <div class="line">This allows the <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-suite-definition">suite definition</a> to be built with python.</div> <div class="line">It also allows communication with the <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-ecflow-server">ecflow_server</a>.</div> </div> <div class="line-block"> <div class="line">This is a very powerful feature, that helps to define very complex suites in a relatively compact way.</div> <div class="line">Consider the following <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-suite">suite</a></div> </div> <div class="highlight-python"><pre>suite test family f1 task a task b task c task d task e endfamily family f2 task a task b task c task d task e endfamily family f3 task a task b task c task d task e endfamily family f4 task a task b task c task d task e endfamily family f5 task a task b task c task d task e endfamily family f6 task a task b task c task d task e endfamily endsuite</pre> </div> This can be written in python as: <div class="highlight-python"><div class="highlight"><pre>def create_suite(name) : suite = ecflow.Suite(name) for i in range(1, 7) : fam = suite.add_family("f" + str(i)) for t in ( "a", "b", "c", "d", "e" ) : fam.add_task(t) return suite </pre></div> </div> <div class="line-block"> <div class="line">Python variables can be used to generate <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-trigger">trigger</a> <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-dependencies">dependencies</a>.</div> <div class="line">Imagine that we want to chain the families f1 to f6, so that f2 runs after f1, f3 after f2 and so on.</div> <div class="line">The following will do the trick:</div> </div> <div class="highlight-python"><div class="highlight"><pre>def create_seqeuntial_suite(name) : suite = ecflow.Suite(name) for i in range(1, 7) : fam = suite.add_family("f" + str(i)) if i != 1: fam.add_trigger("f" + str(i-1) + " == complete") # or fam.add_family( "f%d == complete" % (i-1) ) for t in ( "a", "b", "c", "d", "e" ) : fam.add_task(t) return suite </pre></div> </div> <div class="line-block"> <div class="line">The following python code shows examples of adding the various attributes to a <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-node">node</a> tree.</div> <div class="line">For a detailed explanation please consult the user manual.</div> </div> <div class="highlight-python"><div class="highlight"><pre># provides *examples* of add adding node attributes using the python API # hence does *not* represent a real suite definition from ecflow import * if __name__ == "__main__": # adding variables suite = Suite("s1"); suite.add_variable(Variable("ECF_HOME", "/tmp/")) suite.add_variable("ECF_URL_CMD", "${BROWSER:=firefox} -remote 'openURL(%ECF_URL_BASE%/%ECF_URL%)'") suite.add_variable("NAME", 10) a_dict = { "name":"value", "name2":"value2", "name3":"value3", "name4":"value4" } suite.add_variable(a_dict) # adding limits suite.add_limit( Limit("limitName1", 10) ) suite.add_limit( "limitName3", 10 ) # adding inlimits suite.add_inlimit( InLimit("limitName1", "/s1/f1", 2) ) suite.add_inlimit( "limitName3", "/s1/f1", 2) # add short triggers and complete task = Task("task") task.add_trigger( "t2 == active" ) task.add_complete( "t2 == complete" ) # add long triggers and complete, in example below 'True' mean AND and 'False' means OR task = Task("trigger") task.add_part_trigger( "t1 == complete" ) task.add_part_trigger( "t2 == active", True ) # for long and/or expressions, subsequent expr must be and/or task.add_part_complete( "t3 == complete" ) task.add_part_complete( "t4 == active", False) # for long and/or expressions, subsequent expr must be and/or # add events task.add_event( Event(1) ) task.add_event( 2 ) task.add_event( Event(10, "Eventname") ) task.add_event( 10, "Eventname2" ) task.add_event( "fred" ) # add meter task.add_meter( Meter("metername1", 0, 100, 50) ) task.add_meter( "metername3", 0, 100 ) # add label task.add_label( Label("label_name1", "value") ) task.add_label( "label_name3", "value" ) # add Repeat. A node can only have one repeat, hence we delete the repeat before, adding another task.add_repeat( RepeatInteger("integer", 0, 100, 2) ) task.delete_repeat() task.add_repeat( RepeatEnumerated("enum", ["red", "green", "blue" ] ) ) task.delete_repeat() task.add_repeat( RepeatDate("date", 20100111, 20100115, 2) ) task.delete_repeat() task.add_repeat( RepeatString("string", ["a", "b", "c" ] ) ) task.delete_repeat() # create a time series, used for adding time and today and cron start = TimeSlot(0, 0) finish = TimeSlot(23, 0) incr = TimeSlot(0, 30) time_series = TimeSeries( start, finish, incr, True) # True means relative to suite start # add a today task.add_today( "00:30" ) task.add_today( "+00:30" ) task.add_today( "+00:30 20:00 01:00" ) task.add_today( Today( time_series) ) task.add_today( Today( 0, 10 )) task.add_today( 0, 59, True ) task.add_today( Today(TimeSlot(20, 10)) ) task.add_today( Today(TimeSlot(20, 20), False)) # add time task.add_time( "00:30" ) task.add_time( "+00:30" ) task.add_time( "+00:30 20:00 01:00" ) task.add_time( Time(time_series )) task.add_time( Time( 0, 10 )) task.add_time( 0, 59, True) task.add_time( Time(TimeSlot(20, 10)) ) task.add_time( Time(TimeSlot(20, 20), False)) # add date for i in [ 1, 2, 4, 8, 16 ] : task.add_date( i, 0, 0) # day,month,year, where corresponding 0 means any possible day,month, year task.add_date( Date(1, 1, 2010)) # add day task.add_day( Day(Days.sunday)) task.add_day( Days.monday) task.add_day( "tuesday") # create cron, showing different ways adding the time, to a cron attribute cron = Cron() cron.set_week_days( [0, 1, 2, 3, 4, 5, 6] ) cron.set_days_of_month( [1, 2, 3, 4, 5, 6] ) cron.set_months( [1, 2, 3, 4, 5, 6] ) start = TimeSlot(0, 0) finish = TimeSlot(23, 0) incr = TimeSlot(0, 30) ts = TimeSeries( start, finish, incr, True) # True means relative to suite start cron.set_time_series( ts ) cron1 = Cron() cron1.set_week_days( [0, 1, 2, 3, 4, 5, 6] ) cron1.set_time_series( 1, 30, True ) cron2 = Cron() cron2.set_week_days( [0, 1, 2, 3, 4, 5, 6] ) cron2.set_time_series( "00:30 01:30 00:01" ) cron3 = Cron() cron3.set_week_days( [0, 1, 2, 3, 4, 5, 6] ) cron3.set_time_series( "+00:30" ) # add auto cancel t1 = Task("t1"); t3 = Task("t3") t4 = Task("t4") t5 = Task("t5") t1.add_autocancel( 3 ) # 3 days t3.add_autocancel( 20, 10, True )

and+Clocks

Button Hyperlink

title	Up
type	standard
url	https://confluence.ecmwf.int/display/ECFLOW/Advanced+Topics

Button Hyperlink

title	Next
type	standard
url	https://confluence.ecmwf.int/display/ECFLOW/Indentation

As you have already seen, ecFlow has a ecFlow Python Api: (for both python2 and python3)

import ecflow

This allows the suite definition to be built with python.

It also allows communication with the ecflow_server.

This is a very powerful feature, that helps to define very complex suites in a relatively compact way.
Consider the following suite:

 Code Block
suite test
 family f1
     task a
     task b
     task c
     task d
     task e
 endfamily
 family f2
     task a
     task b
     task c
     task d
     task e
 endfamily
 family f3
     task a
     task b
     task c
     task d
     task e
 endfamily
 family f4
     task a
     task b
     task c
     task d
     task e
 endfamily
 family f5
     task a
     task b
     task c
     task d
     task e
 endfamily
 family f6
     task a
     task b
     task c
     task d
     task e
 endfamily
endsuite
This can be written in python as:

Code Block

language	py

def create_suite(name) : 
    suite = Suite(name)
    for i in range(1, 7) :
        fam = suite.add_family("f" + str(i))
        for t in ( "a", "b", "c", "d", "e" ) :

...

 fam.add_task(t)
    return suite

Code Block

language	py

 def create_suite(name) : 
     return Suite(name,
             [ Family("f{0}".format(i),
                 [ Task(t) for t in ( "a", "b", "c", "d", "e") ]) 
               for i in range(1,7) ])

Python variables can be used to generate trigger dependencies.
Imagine that we want to chain the families f1 to f6, so that f2 runs after f1, f3 after f2 and so on.
The following will do the trick:

Code Block

def create_sequential_suite(name) :
    suite = Suite(name)
    for i in range(1, 7) :
        fam = suite.add_family("f" + str(i))
        if i != 1: 
            fam += Trigger("f" + str(i-1) + " == complete")  # or fam.add_family( "f%d == complete" % (i-1) )
        for t in ( "a", "b", "c", "d", "e" ) :
            fam.add_task(t) 
    return suite

For more detailed example please see the user manual

Adding Node attributes

There are several styles for adding node attributes(Repeat,Time,Today,Date,Day,Cron,Clock,DefStatus,Meter,Event,Variable,Label,Trigger, Complete, Limit,Inlimit,Zombie,Late)

Code Block

language	py

# Functional style
node.add_variable(home,'COURSE')          # c++ style
node.add_limit('limitX',10)               # c++ style

# Using <node>.add(<attributes>) 
node.add(Edit(home=COURSE),               # Notice that add() allows you adjust the indentation
         Limit('limitX',10))              # node.add(<attributes>)  

# in place. When creating a Node, attributes are additional arguments (preferred)
# This also allows indentation.
#   Task(name,<attributes>)
#   Family(name,Node | <attributes>)
#   Suite(name,Node  | <attributes>)
node = Family('t1',                               
           Edit(home='COURSE'),                   
           Limit('limitX',10),
           Task('t1',
              Event('e')))

# Using <node> += <attribute>     adding a single attribute                        
node += Edit(home='COURSE')                            

# Using <node> += [ <attributes> ]  - use list to add multiple attributes
node += [ Edit(home='COURSE'), Limit('limitY',10), Event(1) ]     

# Using node + <attributes>  - A node container(suite | family) must appear on the left hand side. Use brackets to control scope.
node + Edit(home=COURSE) + Limit('limitZ',10)   

# In this example, variable 'name' is added to suite 's/' and not task 't3'     
suite = Suite("s") + Family("f") + Family("f2") + Task("t3") + Edit(name="value")

suite s
  edit name 'value'
  family f
  endfamily
  family f2
  endfamily
  task t3
endsuite


# here we use parenthesis to control where the variable gets added
suite = Suite("s") + Family("f") + Family("f2") + (Task("t3") + Edit(name="value"))

 suite s
  family f
  endfamily
  family f2
  endfamily
  task t3
    edit name 'value'
endsuite

Horizontal Navigation Bar

Button Group

Button Hyperlink

title	Previous
type	standard
url	https://confluence.ecmwf.int/display/ECFLOW/Dates+and+Clocks

Button Hyperlink

title	Up
type	standard
url	https://confluence.ecmwf.int/display/ECFLOW/Advanced+Topics

Button Hyperlink

title	Next
type	standard
url	https://confluence.ecmwf.int/display/ECFLOW/Indentation

t4.add_autocancel( TimeSlot(10, 10), True ) # hour,minutes,relative t5.add_autocancel( Autocancel(1, 10, True) ) # hour,minutes,relative # add late late = Late() late.submitted( TimeSlot(20, 10)) late.active( TimeSlot(20, 10)) late.complete( TimeSlot(20, 10), True) task.add_late( late ) late = Late() late.submitted( 20, 10 ) late.active( 20, 10 ) late.complete( 20, 10, True) t1.add_late( late ) # add defstatus, last one set takes effect task.add_defstatus( DState.complete ) task.add_defstatus( DState.queued ) task.add_defstatus( DState.aborted ) task.add_defstatus( DState.submitted ) task.add_defstatus( DState.suspended ) task.add_defstatus( DState.active ) # add clock clock = Clock(1, 1, 2010, False) # day,month, year, hybrid(true), real(False) clock.set_gain(1, 10, True) # True means positive gain suite = Suite("suite") suite.add_clock(clock) clock = Clock(False) # Use the current time and date with real time clock clock.set_gain_in_seconds(12, True) s1 = Suite("s1") s1.add_clock(clock) </pre></div> </div> </div>

Space shortcuts

Page tree

Versions Compared

Old Version 27

New Version Current

Key

Adding Node attributes