| HTML |
|---|
<div class="section" id="glossary"> <span id="index-0"></span><span id="id1"></span> <dl class="glossary docutils"> <dt id="term-aborted">aborted</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a>.</p> <p class="last">When the ECF_JOB_CMD fails or the <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a> sends a <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –abort <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>, then the task is placed into a aborted state.</p> </dd> <dt id="term-active">active</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a>.</p> <p class="last">If <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> was successful, and <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a> has started, then the <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –init <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> is received by the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> and the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is placed into a active state</p> </dd> <dt id="term-autocancel">autocancel</dt> <dd><p class="first">autocancel is a way to automatically delete a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> which has completed.</p> <p>The delete may be delayed by an amount of time in hours and minutes or expressed in days. Any node may have a single autocancel attribute. If the auto cancelled node is referenced in the <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> expression of other nodes it may leave the node waiting. This can be solved by making sure the <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> expression also checks for the <a class="reference internal" href="#term-unknown"><em class="xref std std-term">unknown</em></a> state. i.e.:</p> <div class="highlight-python"><pre>trigger node_to_cancel == complete or node_to_cancel == unknown</pre> </div> <p>This guards against the ‘node_to_cancel’ being undefined or deleted</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Autocancel" title="ecflow.Autocancel"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Autocancel</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_autocancel" title="ecflow.Node.add_autocancel"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_autocancel</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-autocancel"><tt class="xref std std-token docutils literal"><span class="pre">autocancel</span></tt></a></p> </dd> <dt id="term-check-point">check point</dt> <dd><p class="first">The check point file is like the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>, but includes all the state information.</p> <p>It is periodically saved by the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</p> <p>It can be used to recover the state of the node tree should server die, or machine crash.</p> <p>By default when a <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is started it will look to load the check point file.</p> <p class="last">The default check point file name is <host>.<port>.ecf.check. This can be overridden by the ECF_CHECK environment variable</p> </dd> <dt id="term-child-command">child command</dt> <dd><p class="first">Child command’s(or task requests) are called from within the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> files. They include:</p> <blockquote> <div><table border="1" class="docutils"> <colgroup> <col width="37%" /> <col width="63%" /> </colgroup> <thead valign="bottom"> <tr class="row-odd"><th class="head">Child Command</th> <th class="head">Description</th> </tr> </thead> <tbody valign="top"> <tr class="row-even"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –init</td> <td>Sets the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> to the <a class="reference internal" href="#term-active"><em class="xref std std-term">active</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a></td> </tr> <tr class="row-odd"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –event</td> <td>Set an event</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –meter</td> <td>Change a meter</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –label</td> <td>Change a label</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –wait</td> <td>wait for a expression to evaluate.</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –abort</td> <td>Sets the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> to the <a class="reference internal" href="#term-aborted"><em class="xref std std-term">aborted</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a></td> </tr> <tr class="row-even"><td><a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –complete</td> <td>Sets the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> to the <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a></td> </tr> </tbody> </table> </div></blockquote> <p class="last">The following environment variables must be set for the child commands. ECF_NODE, ECF_NAME ,ECF_PASS and ECF_RID. See <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a>.</p> </dd> <dt id="term-clock">clock</dt> <dd><p class="first">A clock is an attribute of a <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>.</p> <p>A gain can be specified to offset from the given date.</p> <p>The hybrid and real clock’s always runs in phase with the system clock (UTC in UNIX) but can have any offset from the system clock.</p> <p>The clock can be :</p> <blockquote> <div><ul class="simple"> <li><a class="reference internal" href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a></li> <li><a class="reference internal" href="#term-real-clock"><em class="xref std std-term">real clock</em></a></li> <li><a class="reference internal" href="#term-virtual-clock"><em class="xref std std-term">virtual clock</em></a></li> </ul> </div></blockquote> <p><a class="reference internal" href="#term-time"><em class="xref std std-term">time</em></a>, <a class="reference internal" href="#term-day"><em class="xref std std-term">day</em></a> and <a class="reference internal" href="#term-date"><em class="xref std std-term">date</em></a> and <a class="reference internal" href="#term-cron"><em class="xref std std-term">cron</em></a> <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> work a little differently under the clocks.</p> <p>The default clock type is hybrid.</p> <p>If the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is <a class="reference internal" href="#term-shutdown"><em class="xref std std-term">shutdown</em></a> or <a class="reference internal" href="#term-halted"><em class="xref std std-term">halted</em></a> the job <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> is suspended. If this suspension is left for period of time, then it can affect task submission under <strong>hybrid</strong> and <strong>real</strong> clocks. In particular it will affect <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> s with <a class="reference internal" href="#term-time"><em class="xref std std-term">time</em></a>, <a class="reference internal" href="#term-today"><em class="xref std std-term">today</em></a> or <a class="reference internal" href="#term-cron"><em class="xref std std-term">cron</em></a> <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a>.</p> <blockquote> <div><ul> <li><p class="first"><a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> with time series, can result in missed time slots:</p> <div class="highlight-python"><pre>time 10:00 20:00 00:15 # If server is suspended > 15 minutes, time slots can be missed time +00:05 20:00 00:15 # start 5 minutes after the start of the suite, then every 15m until 20:00</pre> </div> </li> <li><p class="first">When the server is placed back into <a class="reference internal" href="#term-running"><em class="xref std std-term">running</em></a> state any time <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> with an expired time slot are submitted straight away. i.e if <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is <a class="reference internal" href="#term-halted"><em class="xref std std-term">halted</em></a> at 10:59 and then placed back into <a class="reference internal" href="#term-running"><em class="xref std std-term">running</em></a> state at 11:20</p> <blockquote> <div><p>time 11:00</p> </div></blockquote> <p>Then any <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> with a expired single time slot dependency will be submitted straight away.</p> </li> </ul> </div></blockquote> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Clock" title="ecflow.Clock"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Clock</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Suite.add_clock" title="ecflow.Suite.add_clock"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Suite.add_clock</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-clock"><tt class="xref std std-token docutils literal"><span class="pre">clock</span></tt></a></p> </dd> <dt id="term-complete">complete</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a>.</p> <p>The node can be set to complete:</p> <blockquote class="last"> <div><p>By the <a class="reference internal" href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a></p> <p>At job end when the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> receives the <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –complete <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a></p> </div></blockquote> </dd> <dt id="term-complete-expression">complete expression</dt> <dd><p class="first">Force a node to be complete <strong>if</strong> the expression evaluates, without running any of the nodes.</p> <p>This allows you to have tasks in the suite which a run only if others fail. In practice the node would need to have a <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> also.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Expression" title="ecflow.Expression"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Expression</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_complete" title="ecflow.Node.add_complete"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_complete</span></tt></a></p> </dd> <dt id="term-cron">cron</dt> <dd><p class="first">Like <a class="reference internal" href="#term-time"><em class="xref std std-term">time</em></a>, cron defines time dependency for a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>, but it can allow the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> to be repeated indefinitely.</p> <p>Note: A cron with a single time dependency will repeat indefinitely</p> <div class="highlight-python"><pre>cron 11:00</pre> </div> <p>When the node becomes complete it will be <a class="reference internal" href="#term-queued"><em class="xref std std-term">queued</em></a> immediately. This means that the suite will never complete, and the output is not directly accessible through <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> <p>If tasks abort, the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> will not schedule it again.</p> <p>If the time the job takes to complete is longer than the interval a “slot” is missed, e.g.</p> <div class="highlight-python"><pre>cron 10:00 20:00 01:00</pre> </div> <p>if the 10:00 run takes more than an hour, the 11:00 run will never occur.</p> <p>If the cron defines months, days of the month, or week days or a single time slot the it relies on a day change, hence if a <a class="reference internal" href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a> is defined, then it will be set to <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a> at the beginning of the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>, without running the corresponding job. Otherwise under a hybrid clock the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> would never <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Cron" title="ecflow.Cron"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Cron</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_cron" title="ecflow.Node.add_cron"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_cron</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-cron"><tt class="xref std std-token docutils literal"><span class="pre">cron</span></tt></a></p> </dd> <dt id="term-date">date</dt> <dd><p class="first">This defines a date dependency for a node.</p> <p>There can be multiple date dependencies. The European format is used for dates, which is: dd.mm.yy as in 31.12.2007. Any of the three number fields can be expressed with a wildcard <cite>*</cite> to mean any valid value. Thus, 01.*.* means the first day of every month of every year.</p> <p>If a <a class="reference internal" href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a> is defined, any node held by a date dependency will be set to <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a> at the beginning of the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>, without running the corresponding job. Otherwise under a hybrid clock the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> would never <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> <p class="last">For python see: <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Date" title="ecflow.Date"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Date</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_date" title="ecflow.Node.add_date"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_date</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-date"><tt class="xref std std-token docutils literal"><span class="pre">date</span></tt></a></p> </dd> <dt id="term-day">day</dt> <dd><p class="first">This defines a day dependency for a node.</p> <p>There can be multiple day dependencies.</p> <p>If a <a class="reference internal" href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a> is defined, any node held by a day dependency will be set to <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a> at the beginning of the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>, without running the corresponding job. Otherwise under a hybrid clock the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> would never <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> <p class="last">For python see: <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Day" title="ecflow.Day"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Day</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_day" title="ecflow.Node.add_day"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_day</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-day"><tt class="xref std std-token docutils literal"><span class="pre">day</span></tt></a></p> </dd> <dt id="term-defstatus">defstatus</dt> <dd><p class="first">Defines the default <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a> for a task/family to be assigned to the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> when the begin command is issued.</p> <p>By default <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> gets queued when you use begin on a <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>. defstatus is useful in preventing suites from running automatically once begun or in setting tasks complete so they can be run selectively.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.DState" title="ecflow.DState"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.DState</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_defstatus" title="ecflow.Node.add_defstatus"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_defstatus</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-defstatus"><tt class="xref std std-token docutils literal"><span class="pre">defstatus</span></tt></a></p> </dd> <dt id="term-dependencies">dependencies</dt> <dd><p class="first">Dependencies are attributes of node, that can suppress/hold a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> from taking part in <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a>.</p> <p>They include <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a>, <a class="reference internal" href="#term-date"><em class="xref std std-term">date</em></a>, <a class="reference internal" href="#term-day"><em class="xref std std-term">day</em></a>, <a class="reference internal" href="#term-time"><em class="xref std std-term">time</em></a>, <a class="reference internal" href="#term-today"><em class="xref std std-term">today</em></a>, <a class="reference internal" href="#term-cron"><em class="xref std std-term">cron</em></a>, <a class="reference internal" href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a>, <a class="reference internal" href="#term-inlimit"><em class="xref std std-term">inlimit</em></a> and <a class="reference internal" href="#term-limit"><em class="xref std std-term">limit</em></a>.</p> <p>A <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> that is dependent can not be started as long as some dependency is holding it or any of its <strong>parent</strong> <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> s.</p> <p class="last">The <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> will check the dependencies every minute, during normal <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> <strong>and</strong> when any <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> causes a state change in the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>.</p> </dd> <dt id="term-directives">directives</dt> <dd><p class="first">Directives start with a % charater. This is referred to as <a class="reference internal" href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a> character. These directives start as the first character on a line in a <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file. The exception is where we use two ECF_MICRO. ie %VAR%, in this case they can occur anywhere on the line. (These directives take part in <a class="reference internal" href="#term-variable-substitution"><em class="xref std std-term">variable substitution</em></a>.) Directives are expanded during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. Examples include:</p> <table border="1" class="last docutils"> <colgroup> <col width="22%" /> <col width="78%" /> </colgroup> <thead valign="bottom"> <tr class="row-odd"><th class="head">Symbol</th> <th class="head">Meaning</th> </tr> </thead> <tbody valign="top"> <tr class="row-even"><td>%include <filename></td> <td>%ECF_INCLUDE% directory is searched for the <tt class="file docutils literal"><span class="pre">filename</span></tt> and the contents included into the job file. If that variable is not defined ECF_HOME is used. This is the recommended format</td> </tr> <tr class="row-odd"><td>%include “filename”</td> <td>Include the contents of the file: %ECF_HOME%/%SUITE%/%FAMILY%/filename into the job.</td> </tr> <tr class="row-even"><td>%include filename</td> <td>Include the contents of the file <tt class="file docutils literal"><span class="pre">filename</span></tt> into the output. The only form that can be used safely must start with a slash ‘/’</td> </tr> <tr class="row-odd"><td>%includenopp filename</td> <td>Same as %include, but the file is not interpreted at all.</td> </tr> <tr class="row-even"><td>%comment</td> <td>Start’s a comment, which is ended by %end directive. The section enclosed by %comment - %end is removed during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a></td> </tr> <tr class="row-odd"><td>%manual</td> <td>Start’s a manual, which is ended by %end directive. The section enclosed by %manual - %end is removed during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. The manual directive is used to create the <a class="reference internal" href="#term-manual-page"><em class="xref std std-term">manual page</em></a> show in <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>.</td> </tr> <tr class="row-even"><td>%nopp</td> <td>Stop pre-processing until a line starting with %end is found. No interpretation of the text will be done( i.e. no variable substitutions)</td> </tr> <tr class="row-odd"><td>%end</td> <td>End processing of %comment or %manual or %nopp</td> </tr> <tr class="row-even"><td>%ecfmicro CHAR</td> <td>Change the directive character, to the character given. If set in an include file the effect is retained for the rest of the job( or until set again). It should be noted that the ecfmicro directive specified in the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file, does <strong>not</strong> effect the variable substitution for ECF_JOB_CMD, ECF_KILL_CMD or ECF_STATUS_CMD variables. They still use <a class="reference internal" href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a>. If no ecfmicro directive exists, we default to using <a class="reference internal" href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a> from the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a></td> </tr> </tbody> </table> </dd> <dt id="term-ecf-file-location-algorithm">ecf file location algorithm</dt> <dd><p class="first"><a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> and job creation checking uses the following algorithm to locate the ‘.ecf’ file corresponding to a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>:</p> <ul> <li><p class="first">ECF_SCRIPT</p> <p>First it uses the generated variable ECF_SCRIPT to locate the script. This variable is generated from: ECF_HOME/<path to task>.ecf</p> <p>Hence if the task path is /suite/f1/f2/t1, then ECF_SCRIPT=ECF_HOME/suite/f1/f2/t1.ecf</p> </li> <li><p class="first">ECF_FILES</p> <p>Second it checks for the user defined ECF_FILES variable. If defined the value of this variable must correspond to a directory. This directory is searched in reverse order.</p> <p>i.e lets assume we have a task: /o/12/fc/model and ECF_FILES is defined as: /home/ecmwf/emos/def/o/ECFfiles</p> <p>The ecFlow will use the following search pattern.</p> <blockquote> <div><ol class="arabic simple"> <li>/home/ecmwf/emos/def/o/ECFfiles/o/12/fc/model.ecf</li> <li>/home/ecmwf/emos/def/o/ECFfiles/12/fc/model.ecf</li> <li>/home/ecmwf/emos/def/o/ECFfiles/fc/model.ecf</li> <li>/home/ecmwf/emos/def/o/ECFfiles/model.ecf</li> </ol> </div></blockquote> </li> <li><p class="first">ECF_HOME</p> <p>Thirdly it searchs for the script in reverse order using ECF_HOME (i.e like ECF_FILES)</p> </li> </ul> <blockquote class="last"> <div><p>If this fails, than the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is placed into the <a class="reference internal" href="#term-aborted"><em class="xref std std-term">aborted</em></a> state. We can check that file can be located before loading the suites into the server.</p> <blockquote> <div><ul class="simple"> <li><a class="reference internal" href="/wiki/display/ECFLOW/Checking+job+creation#checking-job-creation"><em>Checking job creation</em></a></li> <li><a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Defs.check_job_creation" title="ecflow.Defs.check_job_creation"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Defs.check_job_creation</span></tt></a></li> </ul> </div></blockquote> </div></blockquote> </dd> <dt id="term-ecf-script">ecf script</dt> <dd><p class="first">The ecFlow script refers to an ‘.ecf’ file.</p> <p>The script file is transformed into the <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a> by the <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> process.</p> <p>The base name of the script file <strong>must</strong> match its corresponding <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>. i.e t1.ecf , corresponds to the task of name ‘t1’. The script if placed in the ECF_FILES directory, may be re-used by multiple tasks belonging to different families, providing the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> name matches.</p> <p>The ecFlow script is similar to a UNIX shell script.</p> <p class="last">The differences, however, includes the addition of “C” like pre-processing <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a> and ecFlow <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a>‘s. Also the script <em>must</em> include calls to the <strong>init</strong> and <strong>complete</strong> <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> s so that the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is aware when the job starts(i.e changes state to <a class="reference internal" href="#term-active"><em class="xref std std-term">active</em></a>) and finishes ( i.e changes state to <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>)</p> </dd> <dt id="term-ecf-micro">ECF_MICRO</dt> <dd>This is a generated <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a>. The default value is %. This variable is used in <a class="reference internal" href="#term-variable-substitution"><em class="xref std std-term">variable substitution</em></a> during command invocation and default directive character during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. It can be overriden, but must be replaced by a single character.</dd> <dt id="term-ecf-tryno">ECF_TRYNO</dt> <dd><p class="first">This is a generated <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a> that is used in file name generation. It represents the current try number for the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>.</p> <p class="last">After <strong>begin</strong> it is set to 1. The number is advanced if the job is re-run. It is re-set back to 1 after a re-queue. It is used in output and <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a> numbering. (i.e It avoids overwriting the <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a> output during multiple re-runs)</p> </dd> <dt id="term-ecflow">ecFlow</dt> <dd><p class="first">Is the ECMWF work flow manager.</p> <p>A general purpose application designed to schedule a large number of computer process in a heterogeneous environment.</p> <p class="last">Helps computer jobs design, submission and monitoring both in the research and operation departments.</p> </dd> <dt id="term-ecflow-client">ecflow_client</dt> <dd><p class="first">This executable is a command line program; it is used for all communication with the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</p> <p>To see the full range of commands that can be sent to the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> type the following in a UNIX shell:</p> <blockquote> <div>ecflow_client –help</div></blockquote> <p>This functionality is also provided by the <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#client-server-python-api"><em>client server python API</em></a>.</p> <p>The following variables affect the execution of ecflow_client.</p> <p>Since the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> can call ecflow_client( i.e <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>) then typically some are set in an include header. i.e. <a class="reference internal" href="/wiki/display/ECFLOW/Understanding+Includes#head-h"><em>head.h</em></a>.</p> <table border="1" class="last docutils"> <colgroup> <col width="12%" /> <col width="49%" /> <col width="8%" /> <col width="31%" /> </colgroup> <thead valign="bottom"> <tr class="row-odd"><th class="head">Variable Name</th> <th class="head">Explanation</th> <th class="head">Compulsory</th> <th class="head">Example</th> </tr> </thead> <tbody valign="top"> <tr class="row-even"><td>ECF_NODE</td> <td>Name of the host running the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></td> <td>Yes</td> <td>oetzi</td> </tr> <tr class="row-odd"><td>ECF_NAME</td> <td>Path to the task</td> <td>Yes</td> <td>/suite/family/task</td> </tr> <tr class="row-even"><td>ECF_PASS</td> <td>Jobs password.</td> <td>Yes</td> <td>(generated)</td> </tr> <tr class="row-odd"><td>ECF_RID</td> <td>Remote id. Allow easier job kill, and disambiguate a zombie from the real job.</td> <td>Yes</td> <td>(generated)</td> </tr> <tr class="row-even"><td>ECF_PORT</td> <td>Port number of the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></td> <td>No</td> <td>3141.Must match <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></td> </tr> <tr class="row-odd"><td>ECF_TRYNO</td> <td>The number of times the job has run. This is allocated by the server and used in job/output file name generation.</td> <td>No</td> <td>(generated)</td> </tr> <tr class="row-even"><td>ECF_HOSTFILE</td> <td>File that lists alternate hosts to try, if connection to main host fails</td> <td>No</td> <td>/home/user/avi/.ecfhostfile</td> </tr> <tr class="row-odd"><td>ECF_TIMEOUT</td> <td>Maximum time is seconds for the client to deliver message</td> <td>No</td> <td>24*3600 (default value)</td> </tr> <tr class="row-even"><td>ECF_DENIED</td> <td>If server denies client communication and this flag is set, exit with an error. Avoids 24hr hour connection attempt to <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</td> <td>No</td> <td>1</td> </tr> <tr class="row-odd"><td>NO_ECF</td> <td>If set exit’s ecflow_client immediately with success. This allows the scripts to be tested independent of the server</td> <td>No</td> <td>1</td> </tr> </tbody> </table> </dd> <dt id="term-ecflow-server">ecflow_server</dt> <dd><p class="first">This executable is the server.</p> <p>It is responsible for <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> the jobs and responding to <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> requests</p> <p>Multiple servers can be run on the same machine/host providing they are assigned a unique port number.</p> <p>The server record’s all request’s in the log file.</p> <p>The server will periodically(See ECF_CHECKINTERVAL) write out a <a class="reference internal" href="#term-check-point"><em class="xref std std-term">check point</em></a> file.</p> <p>The following environment variables control the execution of the server and may be set before the start of the server. ecflow_server will start happily with out any of these variables being set, since all of them have default values.</p> <table border="1" class="docutils"> <colgroup> <col width="15%" /> <col width="51%" /> <col width="34%" /> </colgroup> <thead valign="bottom"> <tr class="row-odd"><th class="head">Variable Name</th> <th class="head">Explanation</th> <th class="head">Default value</th> </tr> </thead> <tbody valign="top"> <tr class="row-even"><td>ECF_HOME</td> <td>Home for all the <a class="reference internal" href="#term-ecflow"><em class="xref std std-term">ecFlow</em></a> files</td> <td>Current working directory</td> </tr> <tr class="row-odd"><td>ECF_PORT</td> <td>Server port number. Must be unique</td> <td>3141</td> </tr> <tr class="row-even"><td>ECF_LOG</td> <td>History or log file</td> <td><host>.<port>.ecf.log</td> </tr> <tr class="row-odd"><td>ECF_CHECK</td> <td>Name of the checkpoint file</td> <td><host>.<port>.ecf.check</td> </tr> <tr class="row-even"><td>ECF_CHECKOLD</td> <td>Name of the backup checkpoint file</td> <td><host>.<port>.ecf.check.b</td> </tr> <tr class="row-odd"><td>ECF_CHECKINTERVAL</td> <td>Interval in second to save <a class="reference internal" href="#term-check-point"><em class="xref std std-term">check point</em></a> file</td> <td>120</td> </tr> <tr class="row-even"><td>ECF_LISTS</td> <td>White list file. Controls read/write access to the server for each user</td> <td><host>.<port>.ecf.lists</td> </tr> </tbody> </table> <p class="last">The server can be in several states. The default when first started is <a class="reference internal" href="#term-halted"><em class="xref std std-term">halted</em></a>, See <a class="reference internal" href="#term-server-states"><em class="xref std std-term">server states</em></a></p> </dd> <dt id="term-ecflowview">ecflowview</dt> <dd><p class="first">ecflowview executable is the GUI based client, that is used to visualise and monitor the hierarchical structure of the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a></p> <blockquote class="last"> <div><p>state changes in the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>‘s and the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, using colour coding</p> <p>Attributes of the nodes and any <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a></p> <p><a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file and the corresponding <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a></p> </div></blockquote> </dd> <dt id="term-event">event</dt> <dd><p class="first">The purpose of an event is to signal partial completion of a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> and to be able to trigger another job which is waiting for this partial completion.</p> <p>Only tasks can have events and they can be considered as an attribute of a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>.</p> <p>There can be many events and they are displayed as nodes.</p> <p>The event is updated by placing the –event <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> in a <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a>.</p> <p>An event has a number and possibly a name. If it is only defined as a number, its name is the text representation of the number without leading zeroes.</p> <p>For python see: <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Event" title="ecflow.Event"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Event</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_event" title="ecflow.Node.add_event"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_event</span></tt></a> For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-event"><tt class="xref std std-token docutils literal"><span class="pre">event</span></tt></a></p> <p class="last">Events can be referenced in <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> and <a class="reference internal" href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a> s.</p> </dd> <dt id="term-extern">extern</dt> <dd><p class="first">This allows an external <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> to be used in a <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> expression.</p> <p>All <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>‘s in <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a>‘s must be known to <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> by the end of the load command. No cross-suite <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> are allowed unless the names of tasks outside the suite are declared as external. An external <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> reference is considered unknown if it is not defined when the <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> is evaluated. You are strongly advised to avoid cross-suite <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a>.</p> <p class="last">Families and suites that depend on one another should be placed in a single <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>. If you think you need cross-suite dependencies, you should consider merging the suites together and have each as a top-level family in the merged suite. For BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-extern"><tt class="xref std std-token docutils literal"><span class="pre">extern</span></tt></a></p> </dd> <dt id="term-family">family</dt> <dd><p class="first">A family is an organisational entity that is used to provide hierarchy and grouping. It consists of a collection of <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>‘s and families.</p> <p>Typically you place tasks that are related to each other inside the same family, analogous to the way you create directories to contain related files. For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Family" title="ecflow.Family"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Family</span></tt></a>. For BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-family"><tt class="xref std std-token docutils literal"><span class="pre">family</span></tt></a></p> <p class="last">It serves as an intermediate <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> in a <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>.</p> </dd> <dt id="term-halted">halted</dt> <dd>Is a <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> state. See <a class="reference internal" href="#term-server-states"><em class="xref std std-term">server states</em></a></dd> <dt id="term-hybrid-clock">hybrid clock</dt> <dd><p class="first">A hybrid <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a> is a complex notion: the date and time are not connected.</p> <p>The date has a fixed value during the complete execution of the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>. This will be mainly used in cases where the suite does not <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a> in less than 24 hours. This guarantees that all tasks of this suite are using the same <a class="reference internal" href="#term-date"><em class="xref std std-term">date</em></a>. On the other hand, the time follows the time of the machine.</p> <p>Hence the <a class="reference internal" href="#term-date"><em class="xref std std-term">date</em></a> never changes unless specifically altered or unless the suite restarts, either automatically or from a begin command.</p> <p class="last">Under a hybrid <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a> any <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> held by a <a class="reference internal" href="#term-date"><em class="xref std std-term">date</em></a> or <a class="reference internal" href="#term-day"><em class="xref std std-term">day</em></a> dependency will be set to complete at the beginning of the suite. (i.e without its job ever running). Otherwise the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> would never <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> </dd> <dt id="term-inlimit">inlimit</dt> <dd><p class="first">The inlimit works in conjunction with <a class="reference internal" href="#term-limit"><em class="xref std std-term">limit</em></a>/<a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Limit" title="ecflow.Limit"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Limit</span></tt></a> for providing simple load management</p> <p>inlimit is added to the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> that needs to be limited.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.InLimit" title="ecflow.InLimit"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.InLimit</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_inlimit" title="ecflow.Node.add_inlimit"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_inlimit</span></tt></a>. For text BNF see <a class="reference internal" href="#term-inlimit"><em class="xref std std-term">inlimit</em></a></p> </dd> <dt id="term-job-creation">job creation</dt> <dd><p class="first">Job creation or task invocation can be initiated manually via <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a> but also by the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> during <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> when a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> (and <em>all</em> of its parent <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> s) is free of its <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a>.</p> <p>The process of job creation includes:</p> <blockquote> <div><p>o Generating a unique password ECF_PASS, which is placed in <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. See <a class="reference internal" href="/wiki/display/ECFLOW/Understanding+Includes#head-h"><em>head.h</em></a></p> <p>o Locating <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> files , corresponding to the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> in the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>, See <a class="reference internal" href="#term-ecf-file-location-algorithm"><em class="xref std std-term">ecf file location algorithm</em></a></p> <p>o <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a> the contents of the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file</p> </div></blockquote> <p>The steps above transforms an <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> to a <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a> that can be submitted by performing <a class="reference internal" href="#term-variable-substitution"><em class="xref std std-term">variable substitution</em></a> on the ECF_JOB_CMD <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a> and invoking the command.</p> <p>The running jobs will communicate back to the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> by calling <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>‘s.</p> <p class="last">This causes <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a> changes on the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>‘s in the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> and flags can be set to indicate various events.</p> </dd> <dt id="term-job-file">job file</dt> <dd><p class="first">The job file is created by the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> during <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> using the <a class="reference internal" href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a> <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a></p> <p>It is derived from the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> after expanding the pre-processing <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a>.</p> <p>It has the form <task name>.job<<a class="reference internal" href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a>>”, i.e. t1.job1.</p> <p>Note job creation checking will create a job file with an extension with zero. i.e ‘.job0’. See <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Defs.check_job_creation" title="ecflow.Defs.check_job_creation"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Defs.check_job_creation</span></tt></a></p> <p class="last">When the job is run the output file has the <a class="reference internal" href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a> as the extension. i.e t1.1 where ‘t1’ represents the task name and ‘1’ the <a class="reference internal" href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a></p> </dd> <dt id="term-label">label</dt> <dd><p class="first">A label has a name and a value and is a way of <strong>displaying</strong> information in <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> <p>By placing a label <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> s in the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> the user can be informed about progress in <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Label" title="ecflow.Label"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Label</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_label" title="ecflow.Node.add_label"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_label</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-label"><tt class="xref std std-token docutils literal"><span class="pre">label</span></tt></a></p> </dd> <dt id="term-late">late</dt> <dd><p class="first">Define a tag for a node to be late.</p> <p>Suites cannot be late, but you can define a late tag for submitted in a suite, to be inherited by the families and tasks. When a node is classified as being late, the only action <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> takes is to set a flag. <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a> will display these alongside the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> name as an icon (and optionally pop up a window).</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Late" title="ecflow.Late"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Late</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_late" title="ecflow.Node.add_late"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_late</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-late"><tt class="xref std std-token docutils literal"><span class="pre">late</span></tt></a></p> </dd> <dt id="term-limit">limit</dt> <dd><p class="first">Limits provide simple load management by limiting the number of tasks submitted by a specific <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>. Typically you either define limits on <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> level or define a separate suite to hold limits so that they can be used by multiple suites.</p> <p>The limit max value can be changed on the command line</p> <div class="highlight-python"><pre>>ecflow_client --alter change limit_max <limit-name> <new-limit-value> <path-to-limit> >ecflow_client --alter change limit_max limit 2 /suite</pre> </div> <p>It can also be changed in python</p> <div class="highlight-python"><div class="highlight"><pre><span class="c">#!/usr/bin/env python2.7</span> <span class="kn">import</span> <span class="nn">ecflow</span> <span class="k">try</span><span class="p">:</span> <span class="n">ci</span> <span class="o">=</span> <span class="n">ecflow</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span> <span class="n">ci</span><span class="o">.</span><span class="n">alter</span><span class="p">(</span><span class="s">"/suite"</span><span class="p">,</span><span class="s">"change"</span><span class="p">,</span><span class="s">"limit_max"</span><span class="p">,</span><span class="s">"limit"</span><span class="p">,</span> <span class="s">"2"</span><span class="p">)</span> <span class="k">except</span> <span class="ne">RuntimeError</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span> <span class="k">print</span> <span class="s">"Failed: "</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> </pre></div> </div> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Limit" title="ecflow.Limit"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Limit</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_limit" title="ecflow.Node.add_limit"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_limit</span></tt></a>. For BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-limit"><tt class="xref std std-token docutils literal"><span class="pre">limit</span></tt></a> and <a class="reference internal" href="#term-inlimit"><em class="xref std std-term">inlimit</em></a></p> </dd> <dt id="term-manual-page">manual page</dt> <dd><p class="first">Manual pages are part of the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a>.</p> <p>This is to ensure that the manual page is updated when the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> is updated. The manual page is a very important operational tool allowing you to view a description of a task, and possibly describing solutions to common problems. The <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a> can be used to extract the manual page from the script file and is visible in <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>. The manual page is the text contained within the %manual and %end <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a>. They can be seen using the manual button on <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>.</p> <p>The text in the manual page in <strong>not</strong> included in the <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a>.</p> <p>There can be multiple manual sections in the same <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file. When viewed they are simply concatenated. It is good practice to modify the manual pages when the script changes.</p> <p class="last">The manual page may have the %include <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a>.</p> </dd> <dt id="term-meter">meter</dt> <dd><p class="first">The purpose of a meter is to signal proportional completion of a task and to be able to trigger another job which is waiting on this proportional completion.</p> <p>The meter is updated by placing the –meter <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> in a <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a>.</p> <p>For python see: <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Meter" title="ecflow.Meter"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Meter</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_meter" title="ecflow.Node.add_meter"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_meter</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-meter"><tt class="xref std std-token docutils literal"><span class="pre">meter</span></tt></a></p> <p class="last">Meter’s can be referenced in <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> and <a class="reference internal" href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a> expressions.</p> </dd> <dt id="term-node">node</dt> <dd><p class="first"><a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>, <a class="reference internal" href="#term-family"><em class="xref std std-term">family</em></a> and <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> form a hierarchy. Where a <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> serves as the root of the hierarchy. The <a class="reference internal" href="#term-family"><em class="xref std std-term">family</em></a> provides the intermediate nodes, and the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> provide the leaf’s.</p> <p>Collectively <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a>, <a class="reference internal" href="#term-family"><em class="xref std std-term">family</em></a> and <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> can be referred to as nodes.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node" title="ecflow.Node"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node</span></tt></a>.</p> </dd> <dt id="term-pre-processing">pre-processing</dt> <dd><p class="first">Pre-processing takes place during <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> and acts on <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a> specified in <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file.</p> <p>This involves:</p> <blockquote class="last"> <div><p>o expanding any include file <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a>. i.e similar to ‘c’ language pre-processing</p> <p>o removing comments and manual <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a></p> <p>o performing <a class="reference internal" href="#term-variable-substitution"><em class="xref std std-term">variable substitution</em></a></p> </div></blockquote> </dd> <dt id="term-queued">queued</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a>.</p> <p class="last">After the begin command, the task <strong>without</strong> a <a class="reference internal" href="#term-defstatus"><em class="xref std std-term">defstatus</em></a> are placed into the queued state</p> </dd> <dt id="term-real-clock">real clock</dt> <dd>A <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> using a real <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a> will have its <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a> matching the clock of the machine. Hence the <a class="reference internal" href="#term-date"><em class="xref std std-term">date</em></a> advances by one day at midnight.</dd> <dt id="term-repeat">repeat</dt> <dd><p class="first">Repeats provide looping functionality. There can only be a single repeat on a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>.</p> <blockquote> <div><p>repeat day step [ENDDATE] # only for suites</p> <p>repeat integer VARIABLE start end [step]</p> <p>repeat enumerated VARIABLE first [second [third ...]]</p> <p>repeat string VARIABLE str1 [str2 ...]</p> <p>repeat file VARIABLE filename</p> <p>repeat date VARIABLE yyyymmdd yyyymmdd [delta]</p> </div></blockquote> <p>The repeat VARIABLE can be used in <a class="reference internal" href="#term-trigger"><em class="xref std std-term">trigger</em></a> and <a class="reference internal" href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a> expressions.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_repeat" title="ecflow.Node.add_repeat"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_repeat</span></tt></a>, <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Repeat" title="ecflow.Repeat"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Repeat</span></tt></a>, <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatDate" title="ecflow.RepeatDate"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.RepeatDate</span></tt></a>, <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatEnumerated" title="ecflow.RepeatEnumerated"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.RepeatEnumerated</span></tt></a>, <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatInteger" title="ecflow.RepeatInteger"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.RepeatInteger</span></tt></a>, <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatDay" title="ecflow.RepeatDay"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.RepeatDay</span></tt></a> For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-repeat"><tt class="xref std std-token docutils literal"><span class="pre">repeat</span></tt></a></p> </dd> <dt id="term-running">running</dt> <dd>Is a <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> state. See <a class="reference internal" href="#term-server-states"><em class="xref std std-term">server states</em></a></dd> <dt id="term-scheduling">scheduling</dt> <dd><p class="first">The <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is responsible for <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> scheduling.</p> <p class="last">It will check <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> in the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> every minute. If these <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> are free, the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> will submit the task. See <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a>.</p> </dd> <dt id="term-server-states">server states</dt> <dd><p class="first">The following tables reflects the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> capabilities in the different states</p> <table border="1" class="last docutils"> <colgroup> <col width="22%" /> <col width="16%" /> <col width="16%" /> <col width="19%" /> <col width="26%" /> </colgroup> <thead valign="bottom"> <tr class="row-odd"><th class="head">State</th> <th class="head">User Request</th> <th class="head">Task Request</th> <th class="head">Job Scheduling</th> <th class="head">Auto-Check-pointing</th> </tr> </thead> <tbody valign="top"> <tr class="row-even"><td><a class="reference internal" href="#term-running"><em class="xref std std-term">running</em></a></td> <td>yes</td> <td>yes</td> <td>yes</td> <td>yes</td> </tr> <tr class="row-odd"><td><a class="reference internal" href="#term-shutdown"><em class="xref std std-term">shutdown</em></a></td> <td>yes</td> <td>yes</td> <td>no</td> <td>yes</td> </tr> <tr class="row-even"><td><a class="reference internal" href="#term-halted"><em class="xref std std-term">halted</em></a></td> <td>yes</td> <td>no</td> <td>no</td> <td>no</td> </tr> </tbody> </table> </dd> <dt id="term-shutdown">shutdown</dt> <dd>Is a <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> state. See <a class="reference internal" href="#term-server-states"><em class="xref std std-term">server states</em></a></dd> <dt id="term-status">status</dt> <dd><p class="first">Each <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> in <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> has a status.</p> <p>Status reflects the state of the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>. In <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a> the background colour of the text reflects the status.</p> <p><a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> status are: <a class="reference internal" href="#term-unknown"><em class="xref std std-term">unknown</em></a>, <a class="reference internal" href="#term-queued"><em class="xref std std-term">queued</em></a>, <a class="reference internal" href="#term-submitted"><em class="xref std std-term">submitted</em></a>, <a class="reference internal" href="#term-active"><em class="xref std std-term">active</em></a>, <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>, <a class="reference internal" href="#term-aborted"><em class="xref std std-term">aborted</em></a> and <a class="reference internal" href="#term-suspended"><em class="xref std std-term">suspended</em></a></p> <p class="last"><a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> status are: <a class="reference internal" href="#term-shutdown"><em class="xref std std-term">shutdown</em></a>, <a class="reference internal" href="#term-halted"><em class="xref std std-term">halted</em></a>, <a class="reference internal" href="#term-running"><em class="xref std std-term">running</em></a> this is shown on the root node in <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> </dd> <dt id="term-submitted">submitted</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a>.</p> <p class="last">When the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> <a class="reference internal" href="#term-dependencies"><em class="xref std std-term">dependencies</em></a> are resolved/free the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> places the task into a submitted state. However if the ECF_JOB_CMD fails, the task is placed into the <a class="reference internal" href="#term-aborted"><em class="xref std std-term">aborted</em></a> state</p> </dd> <dt id="term-suite">suite</dt> <dd><p class="first">A suite is organisational entity. It is serves as the root <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> in a <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>. It should be used to hold a set of jobs that achieve a common function. It can be used to hold user <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a> s that are common to all of its children.</p> <p>Only a suite node can have a <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a>.</p> <p class="last">It is a collection of <a class="reference internal" href="#term-family"><em class="xref std std-term">family</em></a>‘s, <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a>‘s, <a class="reference internal" href="#term-repeat"><em class="xref std std-term">repeat</em></a> and a single <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a> definition. For a complete list of attributes look at BNF for <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-suite"><tt class="xref std std-token docutils literal"><span class="pre">suite</span></tt></a>. For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Suite" title="ecflow.Suite"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Suite</span></tt></a>.</p> </dd> <dt id="term-suite-definition">suite definition</dt> <dd><p class="first">The suite definition is the hierarchical <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> tree.</p> <p>It describes how your <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>‘s run and interact.</p> <p>It can built up using:</p> <blockquote> <div><ul> <li><p class="first">Ascii text file by following the rules defined in the ecFlow <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammer"><em>Definition file Grammar</em></a>.</p> <p>Hence any language can be used, to generate this format.</p> </li> <li><p class="first"><a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#suite-definition-python-api"><em>suite definition python API</em></a></p> </li> </ul> </div></blockquote> <p class="last">Once the definition is built, it can be loaded into the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, and started. It can be monitored by <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> </dd> <dt id="term-suspended">suspended</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> state. A <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> can be placed into the suspended state via a <a class="reference internal" href="#term-defstatus"><em class="xref std std-term">defstatus</em></a> or via <a class="reference internal" href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> <p class="last">A suspended <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> including any of its children can not take part in <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> until the node is resumed.</p> </dd> <dt id="term-task">task</dt> <dd><p class="first">A task represents a job that needs to be carried out. It serves as a leaf <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> in a <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a></p> <p>Only tasks can be submitted.</p> <p>A job inside a task <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> should generally be re-entrant so that no harm is done by rerunning it, since a task may be automatically submitted more than once if it aborts.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Task" title="ecflow.Task"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Task</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-task"><tt class="xref std std-token docutils literal"><span class="pre">task</span></tt></a></p> </dd> <dt id="term-time">time</dt> <dd><p class="first">This defines a time dependency for a node.</p> <p>Time is expressed in the format [h]h:mm. Only numeric values are allowed. There can be multiple time dependencies for a node, but overlapping times may cause unexpected results. To define a series of times, specify the start time, end time and a time increment. If the start time begins with ‘+’, times are relative to the beginning of the suite or, in repeated families, relative to the beginning of the repeated family.</p> <p>If the time the job takes to complete is longer than the interval a “slot” is missed, e.g.</p> <div class="highlight-python"><pre>time 10:00 20:00 01:00</pre> </div> <p>if the 10:00 run takes more than an hour, the 11:00 run will never occur.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Time" title="ecflow.Time"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Time</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_time" title="ecflow.Node.add_time"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_time</span></tt></a>. For BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-time"><tt class="xref std std-token docutils literal"><span class="pre">time</span></tt></a></p> </dd> <dt id="term-today">today</dt> <dd><p class="first">Like <a class="reference internal" href="#term-time"><em class="xref std std-term">time</em></a>, but If the suites begin time is <strong>past</strong> the time given for the “today” command the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> is free to run (as far as the time dependency is concerned).</p> <p>For example</p> <div class="highlight-python"><pre>task x today 10:00</pre> </div> <p>If we begin or re-queue the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> at 9.00 am, then the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> in held until 10.00 am. However if we begin or re-queue the suite at 11.00am, the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is run immediately.</p> <p>No lets look at time</p> <div class="highlight-python"><pre>task x time 10:00</pre> </div> <p>If we begin or re-queue the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> at 9.00am, then the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> in held until 10.00 am. If we begin or re-queue the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> at 11.00am, the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is still held.</p> <p>If the time the job takes to complete is longer than the interval a “slot” is missed, e.g.</p> <div class="highlight-python"><pre>today 10:00 20:00 01:00</pre> </div> <p>if the 10:00 run takes more than an hour, the 11:00 run will never occur.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Today" title="ecflow.Today"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Today</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-today"><tt class="xref std std-token docutils literal"><span class="pre">today</span></tt></a></p> </dd> <dt id="term-trigger">trigger</dt> <dd><p class="first">Triggers defines a dependency for a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> or <a class="reference internal" href="#term-family"><em class="xref std std-term">family</em></a>.</p> <p>There can be only one trigger dependency per <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>, but that can be a complex boolean expression of the <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a> of several nodes. Triggers should be avoided on suites. A node with a trigger can only be activated when its trigger has expired. A trigger holds the node as long as the trigger’s expression evaluation returns false.</p> <p>The keywords in trigger expressions are: <a class="reference internal" href="#term-unknown"><em class="xref std std-term">unknown</em></a>, <a class="reference internal" href="#term-suspended"><em class="xref std std-term">suspended</em></a>, <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>, <a class="reference internal" href="#term-queued"><em class="xref std std-term">queued</em></a>, <a class="reference internal" href="#term-submitted"><em class="xref std std-term">submitted</em></a>, <a class="reference internal" href="#term-active"><em class="xref std std-term">active</em></a>, <a class="reference internal" href="#term-aborted"><em class="xref std std-term">aborted</em></a> and <strong>clear</strong> and <strong>set</strong> for <a class="reference internal" href="#term-event"><em class="xref std std-term">event</em></a> status.</p> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Expression" title="ecflow.Expression"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Expression</span></tt></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_trigger" title="ecflow.Node.add_trigger"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_trigger</span></tt></a></p> </dd> <dt id="term-unknown">unknown</dt> <dd><p class="first">Is a <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a>.</p> <p class="last">This is the default <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> <a class="reference internal" href="#term-status"><em class="xref std std-term">status</em></a> when a <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> is loaded into the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></p> </dd> <dt id="term-user-commands">user commands</dt> <dd>User commands are any client to server requests that are <strong>not</strong> <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> s.</dd> <dt id="term-variable">variable</dt> <dd><p class="first">ecFlow makes heavy use of different kinds of variables.There are several kinds of variables:</p> <blockquote> <div><p>Environment variables: which are set in the UNIX shell before the <a class="reference internal" href="#term-ecflow"><em class="xref std std-term">ecFlow</em></a> starts. These control <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, and <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> .</p> <p>suite definition variables: Also referred to as user variables. These control <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, and <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> and are available for use in <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a>.</p> <p>Generated variables: These are generated within the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> node tree during <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> and are available for use in the <a class="reference internal" href="#term-job-file"><em class="xref std std-term">job file</em></a>.</p> </div></blockquote> <p class="last">For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_variable" title="ecflow.Node.add_variable"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.Node.add_variable</span></tt></a>. For BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-variable"><tt class="xref std std-token docutils literal"><span class="pre">variable</span></tt></a></p> </dd> <dt id="term-variable-inheritance">variable inheritance</dt> <dd><p class="first">When a <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a> is needed at <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> time, it is first sought in the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> itself.</p> <p>If it is not found in the <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a>, it is sought from the task’s parent and so on, up through the <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a> levels until found.</p> <p>For any <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>, there are two places to look for variables.</p> <p class="last">Suite definition variables are looked for first, and then any generated variables.</p> </dd> <dt id="term-variable-substitution">variable substitution</dt> <dd><p class="first">Takes place during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a> <em>or</em> command invocation.(i.e ECF_JOB_CMD,ECF_KILL_CMD,etc)</p> <p>It involves searching each line of <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file or command, for <a class="reference internal" href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a> character. typically ‘%’</p> <p>The text between two % character, defines a variable. i.e %VAR%</p> <p>This variable is searched for in the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>.</p> <p>First the suite definition variables( sometimes referred to as user variables) are searched and then Repeat variable name, and finally the generated variables.If no variable is found then the same search pattern is repeated up the node tree.</p> <p>The value of the <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a> is replaced between the % characters.</p> <p>If the variable is not found in the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> during pre-processing then <a class="reference internal" href="#term-job-creation"><em class="xref std std-term">job creation</em></a> fails, and an error message is written to the log file, and the task is placed into the <a class="reference internal" href="#term-aborted"><em class="xref std std-term">aborted</em></a> state.</p> <p>To avoid this, variables in the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> can be defined as:</p> <blockquote class="last"> <div>%VAR:replacement% : This is similar to %VAR% but if VAR is not found in the <a class="reference internal" href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> then ‘replacement’ is used.</div></blockquote> </dd> <dt id="term-virtual-clock">virtual clock</dt> <dd><p class="first">Like <a class="reference internal" href="#term-real-clock"><em class="xref std std-term">real clock</em></a> until the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is suspended (i.e <a class="reference internal" href="#term-shutdown"><em class="xref std std-term">shutdown</em></a> or <a class="reference internal" href="#term-halted"><em class="xref std std-term">halted</em></a>), the suites <a class="reference internal" href="#term-clock"><em class="xref std std-term">clock</em></a> is also suspended.</p> <p>Hence will honour relative times in <a class="reference internal" href="#term-cron"><em class="xref std std-term">cron</em></a>, <a class="reference internal" href="#term-today"><em class="xref std std-term">today</em></a> and <a class="reference internal" href="#term-time"><em class="xref std std-term">time</em></a> dependencies. It is possible to have a combination of hybrid/real and virtual.</p> <p class="last">More useful when we want complete adherence to time related dependencies at the expense being out of sync with system time.</p> </dd> <dt id="term-zombie">zombie</dt> <dd><p class="first">Zombies are running jobs that fail authentication when communicating with the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></p> <p><a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> s like (init, event,meter, label, abort,complete) are placed in the <a class="reference internal" href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file and are used to communicate with the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</p> <p>The <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> authenticates each connection attempt made by the <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>. Authentication can fail for a number of reasons:</p> <blockquote> <div><ul class="simple"> <li>password(ECF_PASS) supplied with the <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>, does not match the one in the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li>path name(ECF_NAME) supplied with the <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>, does not locate a <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> in the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li>process id(ECF_RID) supplied with <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>, does not correspond with the one stored in the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li><a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is already <a class="reference internal" href="#term-active"><em class="xref std std-term">active</em></a>, but receives another init <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a></li> <li><a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is already <a class="reference internal" href="#term-complete"><em class="xref std std-term">complete</em></a>, but receives another <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a></li> </ul> </div></blockquote> <p>When authentication fails the job is considered to be a zombie. The <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> will keep a note of the zombie for a period of time, before it is automatically removed. However the removed zombie, may well re-appear. ( this is because each <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a> will continue attempting to contact the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> for 24 hours. This is configurable see ECF_TIMEOUT on <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a>)</p> <p>For python see <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr" title="ecflow.ZombieAttr"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.ZombieAttr</span></tt></a>, <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieUserActionType" title="ecflow.ZombieUserActionType"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.ZombieUserActionType</span></tt></a></p> <p class="last">There are several types of zombies see <a class="reference internal" href="#term-zombie-type"><em class="xref std std-term">zombie type</em></a> and <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieType" title="ecflow.ZombieType"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.ZombieType</span></tt></a></p> </dd> <dt id="term-zombie-attribute">zombie attribute</dt> <dd><p class="first">The zombie attribute defines how a <a class="reference internal" href="#term-zombie"><em class="xref std std-term">zombie</em></a> should be handled in an automated fashion. Very careful consideration should be taken before this attribute is added as it may hide a genuine problem. It can be added to any <a class="reference internal" href="#term-node"><em class="xref std std-term">node</em></a>. But is best defined at the <a class="reference internal" href="#term-suite"><em class="xref std std-term">suite</em></a> or <a class="reference internal" href="#term-family"><em class="xref std std-term">family</em></a> level. If there is no zombie attribute the default behaviour is to block the <a class="reference internal" href="#term-child-command"><em class="xref std std-term">child command</em></a>.</p> <p class="last">To add a zombie attribute in python, please see: <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr" title="ecflow.ZombieAttr"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.ZombieAttr</span></tt></a></p> </dd> <dt id="term-zombie-type">zombie type</dt> <dd><p class="first">See <a class="reference internal" href="#term-zombie"><em class="xref std std-term">zombie</em></a> and class <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr" title="ecflow.ZombieAttr"><tt class="xref py py-class docutils literal"><span class="pre">ecflow.ZombieAttr</span></tt></a> for further information. There are several types of zombies:</p> <blockquote> <div><ul> <li><dl class="first docutils"> <dt>path</dt> <dd><ul class="first last simple"> <li>The task path can not be found in the server, because node tree was deleted, replaced,reload, server crashed or backup server does not have node tree.</li> <li>Jobs could have been created, via server <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> or by <a class="reference internal" href="#term-user-commands"><em class="xref std std-term">user commands</em></a></li> </ul> </dd> </dl> </li> <li><dl class="first docutils"> <dt>user</dt> <dd><p class="first last">Job is created by <a class="reference internal" href="#term-user-commands"><em class="xref std std-term">user commands</em></a> like, rerun, re-queue. User zombies are differentiated from server(scheduled) since they are automatically created when the force option is used and we have tasks in an <a class="reference internal" href="#term-active"><em class="xref std std-term">active</em></a> or <a class="reference internal" href="#term-submitted"><em class="xref std std-term">submitted</em></a> states.</p> </dd> </dl> </li> <li><dl class="first docutils"> <dt>ecf</dt> <dd><p class="first">Jobs are created as part of the normal <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a></p> <ul class="last simple"> <li>Server crashed ( or terminated and restarted) and the recovered <a class="reference internal" href="#term-check-point"><em class="xref std std-term">check point</em></a> file is out of date.</li> <li>A <a class="reference internal" href="#term-task"><em class="xref std std-term">task</em></a> is repeatedly re-run, earlier copies will not be remembered.</li> <li>Job sent by another <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, but which can not talk to the original <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li>Network glitches/network down</li> </ul> </dd> </dl> </li> </ul> </div></blockquote> <p class="last">The type of the zombie is not fixed and may change.</p> </dd> </dl> </div> |
Page History
Overview
Community Forums
Content Tools