ecFlow's documentation is now on readthedocs!

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

Currently in ecflow we can have jobs that are identical but vary only in the step.

In these cases each step carries a latency, i.e submission  cost,(i.e. starting hundreds of new processes)

The Queue attribute was created to address this situation.

suite test_queue
      family f1
          queue q1 1 2 3 4 5 6 7
          task t
      endfamily
      family f2
          task a
             queue q2 1 2 3
          task b
             trigger /test_queue/f1:q1 > 1      # notice that queue name is accessible to the trigger
          task c
             trigger /test_queue/f2/a:q2 > 1
       endfamily
endsuite

There is a new child command --queue, that will  signal when a step is active,complete or has aborted.

New Child command, for use in .ecf scripts
# Note: because --queue is treated like a child command(init,complete,event,label,meter,abort,wait), the task path ECF_NAME is read from the environment

# The --queue command will search up the node hierarchy for the queue name. If not found it fails.

step=$(ecflow_client --queue queue_name  active)                # returns first queued/aborted step from the server and makes it active, Return "NULL" for the last step.
ecflow_client --queue queue_name complete $step                 # Tell the server that step has completed for the given queue
ecflow_client --queue queue_name aborted  $step                 # Tell the server that step has aborted for the given queue
no_of_aborted=$(ecflow_client --queue queue_name no_of_aborted) # returns as a string the number of aborted steps 
ecflow_client --queue queue_name reset                          # sets the index to the first queued/aborted step. Allows steps to be reprocessed for errors

Heres how it could be used in a script:

step=""
QNAME="my_queue_name"
while [1 == 1 ] ; do
   # this return the first queued/aborted step, then increments to next step, return <NULL> when all steps processed
   step=$(ecflow_client --queue=$QNAME active) # of the form string  i.e \"003\". this step is now active
   if [[ $step == "<NULL>" ]] ; then
        break; # no more steps
   fi
   ...
   ecflow_client --queue=$QNAME $step complete   # tell ecflow this step completed
done
            
trap() { ecflow_client --queue=$QNAME $step aborted # tell ecflow this step failed }


This attribute makes it possible to follow a producer(server)/consumer(tasks) pattern. Note additional task consumers can be added for load balancing.

  • No labels