Currently, in ecFlow, we can have jobs that are identical but vary only in the step.
In these cases, each step carries a latency, i.e. submission cost,(i.e. starting hundreds of new processes)
The Queue attribute was created to address this situation.
suite test
family f1
task 001
task 002
task 003
task 004
task 005
endfamily
endsuite
| suite test
family f1
queue q1 001 002 003 004 005 006
task t
endfamily
endsuite
|
|---|
suite test_queue
family f1
queue q1 001 002 003 004 005 006 007
task t
endfamily
family f2
queue q2 1 2 3 4 5 6 8 9 10
task a
task b
# notice that queue name is accessible to the trigger
trigger /test_queue/f1:q1 > 5
task c
trigger ../f2/a:q2 > 9
endfamily
endsuite
There is a new child command --queue, that will signal when a step is active, complete, or has aborted.
New Child command, for use in .ecf scripts
# Note: because --queue is treated like a child command(init,complete,event,label,meter,abort,wait), the task path ECF_NAME is read from the environment # The --queue command will search up the node hierarchy for the queue name. If not found it fails. step=$(ecflow_client --queue queue_name active) # returns first queued/aborted step from the server and makes it active, Return "NULL" for the last step. ecflow_client --queue queue_name complete $step # Tell the server that step has completed for the given queue ecflow_client --queue queue_name aborted $step # Tell the server that step has aborted for the given queue no_of_aborted=$(ecflow_client --queue queue_name no_of_aborted) # returns as a string the number of aborted steps ecflow_client --queue queue_name reset # sets the index to the first queued/aborted step. Allows steps to be reprocessed for errors
This attribute makes it possible to follow a producer(server)/consumer(tasks) pattern. Note additional task consumers can be added for load balancing.
The queue values can be strings, however, if they are to be used in trigger expressions, they must be convertible to integers.
suite test_queue
family f1
queue q1 red orange yellow green blue indigo violet
task t
endfamily
endsuite