Currently, in ecFlow, we can have jobs that are identical but vary only in the step.
In these cases, each step carries a latency, i.e. submission cost,(i.e. starting hundreds of new processes)
The Queue attribute was created to address this situation.
suite test_queue
family f1
queue q1 1 2 3 4 5 6 7
task t
endfamily
family f2
task a
queue q2 1 2 3 4 5 6 8 9 10
task b
# notice that queue name is accessible to the trigger
trigger /test_queue/f1:q1 > 5
task c
trigger /test_queue/f2/a:q2 > 9
endfamily
endsuite
There is a new child command --queue, that will signal when a step is active, complete, or has aborted.
New Child command, for use in .ecf scripts
# Note: because --queue is treated like a child command(init,complete,event,label,meter,abort,wait), the task path ECF_NAME is read from the environment # The --queue command will search up the node hierarchy for the queue name. If not found it fails. step=$(ecflow_client --queue queue_name active) # returns first queued/aborted step from the server and makes it active, Return "NULL" for the last step. ecflow_client --queue queue_name complete $step # Tell the server that step has completed for the given queue ecflow_client --queue queue_name aborted $step # Tell the server that step has aborted for the given queue no_of_aborted=$(ecflow_client --queue queue_name no_of_aborted) # returns as a string the number of aborted steps ecflow_client --queue queue_name reset # sets the index to the first queued/aborted step. Allows steps to be reprocessed for errors
This attribute makes it possible to follow a producer(server)/consumer(tasks) pattern. Note additional task consumers can be added for load balancing.