Currently, in ecflow ecFlow, we can have jobs that have are identical but vary only in the step.
In these cases, each step carries a latency, i.e. submission cost,(i.e. starting hundreds of new processes)
The Queue attribute was created to address this situation.
...
...
...
...
...
...
...
...
...
...
...
...
004
task 005
endfamily
endsuite |
| |
---|
...
family f1
queue q1 001 002 003 004 005 006
task t
endfamily
endsuite |
|
---|
Code Block |
---|
suite test_queue
t
endfamily
family f2f1
queue q1 001 002 003 004 005 task006 a007
task t
endfamily
family f2
queue q2 1 2 3 4 5 6 8 9 10
task a
task b
# notice that queue name is accessible to the trigger
trigger /test_queue/f1:q1 > 1
5
task c
trigger /test_queue../f2/a:q2 > 19
endfamily
endsuite |
There are is a new child commandscommand --queue, that are used to will signal when a step is active, complete, or has aborted.
Code Block |
---|
language | bash |
---|
title | New Child command, for use in .ecf scripts |
---|
|
# Note: because --queue is treated like a child command(init,complete,event,label,meter,abort,wait), the task path ECF_NAME is read from the environment
# The --queue command will search up the node hierarchy for the queue name. If not found it fails.
step=$(ecflow_client --queue queue_name active) # Thisreturns willfirst returnqueued/aborted next step formfrom the server and makemakes it active, Return "NULL" for the last step.
ecflow_client --queue queue_name complete $step # completeTell the currentserver activethat step has completed for the given queue
ecflow_client --queue queue_name aborted $step # Tell tellthe server that step has aborted for the given queue
no_of_aborted=$(ecflow_client --queue queue_name no_of_aborted) # returns as a string the number of aborted steps
ecflow_client --queue queue_name reset # sets the index to the first queued/aborted step. Allows steps to be reprocessed for errors |
This attribute makes it possible to follow a producer(server)/consumer(tasks) pattern. Note additional task consumers can be added for load balancing.
The queue values can be strings, however, if they are to be used in trigger expressions, they must be convertible to integers.
Code Block |
---|
suite test_queue
family f1
queue q1 red orange yellow green blue indigo violet
task t
endfamily
endsuite current step failed. |