| HTML |
|---|
<div id="glossary">
<span id="index-0"></span><span id="id1"></span>
<dl >
<dt id="term-aborted">aborted</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> <a
href="#term-status"><em >status</em></a>.</p>
<p >When the ECF_JOB_CMD fails or the <a
href="#term-job-file"><em >job file</em></a> sends a
<a href="#term-ecflow-client"><em class="xref std
std-term">ecflow_client</em></a> –abort <a
href="#term-child-command"><em >child
command</em></a>, then
the task is placed into a aborted state.</p>
</dd>
<dt id="term-active">active</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> <a
href="#term-status"><em >status</em></a>.</p>
<p >If <a href="#term-job-creation"><em
>job creation</em></a> was successful, and <a
href="#term-job-file"><em class="xref std
std-term">job file</em></a> has started, then the <a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–init
<a href="#term-child-command"><em class="xref std
std-term">child command</em></a> is received by the <a class="reference
internal" href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> and the <a
href="#term-task"><em >task</em></a> is placed into a
active state</p>
</dd>
<dt id="term-autocancel">autocancel</dt>
<dd><p >autocancel is a way to automatically delete a <a
href="#term-node"><em class="xref std
std-term">node</em></a> which has completed.</p>
<p>The delete may be delayed by an amount of time in hours and minutes or
expressed in days. Any node may have a single autocancel attribute.
If the auto cancelled node is referenced in the <a
href="#term-trigger"><em >trigger</em></a> expression
of other nodes
it may leave the node waiting. This can be solved by making sure the <a
href="#term-trigger"><em class="xref std
std-term">trigger</em></a>
expression also checks for the <a
href="#term-unknown"><em >unknown</em></a> state.
i.e.:</p>
<div ><pre>trigger node_to_cancel == complete or
node_to_cancel == unknown</pre>
</div>
<p>This guards against the ‘node_to_cancel’ being undefined or
deleted</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Autocancel"
title="ecflow.Autocancel"><tt ><span
>ecflow.Autocancel</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_autocancel"
title="ecflow.Node.add_autocancel"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_autocancel</span></tt></a>. For text
BNF see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-autocancel"><tt
><span
>autocancel</span></tt></a></p>
</dd>
<dt id="term-check-point">check point</dt>
<dd><p >The check point file is like the <a class="reference
internal" href="#term-suite-definition"><em >suite
definition</em></a>, but includes all the state information.</p>
<p>It is periodically saved by the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>.</p>
<p>It can be used to recover the state of the node tree should server die, or
machine crash.</p>
<p>By default when a <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
is started it will look to load the check point file.</p>
<p >The default check point file name is
<host>.<port>.ecf.check. This can be overridden by the ECF_CHECK
environment variable</p>
</dd>
<dt id="term-child-command">child command</dt>
<dd><p >Child command’s(or task requests) are called from
within the <a href="#term-ecf-script"><em
>ecf script</em></a> files. They include:</p>
<blockquote>
<div><table border="1" >
<colgroup>
<col width="37%" />
<col width="63%" />
</colgroup>
<thead valign="bottom">
<tr ><th >Child Command</th>
<th >Description</th>
</tr>
</thead>
<tbody valign="top">
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–init</td>
<td>Sets the <a href="#term-task"><em class="xref
std std-term">task</em></a> to the <a
href="#term-active"><em >active</em></a> <a
href="#term-status"><em class="xref std
std-term">status</em></a></td>
</tr>
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–event</td>
<td>Set an event</td>
</tr>
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–meter</td>
<td>Change a meter</td>
</tr>
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–label</td>
<td>Change a label</td>
</tr>
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–wait</td>
<td>wait for a expression to evaluate.</td>
</tr>
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–abort</td>
<td>Sets the <a href="#term-task"><em class="xref
std std-term">task</em></a> to the <a
href="#term-aborted"><em >aborted</em></a> <a
href="#term-status"><em class="xref std
std-term">status</em></a></td>
</tr>
<tr ><td><a
href="#term-ecflow-client"><em >ecflow_client</em></a>
–complete</td>
<td>Sets the <a href="#term-task"><em class="xref
std std-term">task</em></a> to the <a
href="#term-complete"><em >complete</em></a> <a
href="#term-status"><em class="xref std
std-term">status</em></a></td>
</tr>
</tbody>
</table>
</div></blockquote>
<p >The following environment variables must be set for the child
commands. ECF_NODE, ECF_NAME ,ECF_PASS and
ECF_RID. See <a href="#term-ecflow-client"><em
>ecflow_client</em></a>.</p>
</dd>
<dt id="term-clock">clock</dt>
<dd><p >A clock is an attribute of a <a
href="#term-suite"><em >suite</em></a>.</p>
<p>A gain can be specified to offset from the given date.</p>
<p>The hybrid and real clock’s always runs in phase with the system clock
(UTC in UNIX)
but can have any offset from the system clock.</p>
<p>The clock can be :</p>
<blockquote>
<div><ul >
<li><a href="#term-hybrid-clock"><em class="xref std
std-term">hybrid clock</em></a></li>
<li><a href="#term-real-clock"><em class="xref std
std-term">real clock</em></a></li>
<li><a href="#term-virtual-clock"><em class="xref
std std-term">virtual clock</em></a></li>
</ul>
</div></blockquote>
<p><a href="#term-time"><em class="xref std
std-term">time</em></a>, <a href="#term-day"><em
>day</em></a> and <a
href="#term-date"><em >date</em></a> and <a
href="#term-cron"><em class="xref std
std-term">cron</em></a> <a
href="#term-dependencies"><em >dependencies</em></a>
work a little differently under the clocks.</p>
<p>The default clock type is hybrid.</p>
<p>If the <a href="#term-ecflow-server"><em
>ecflow_server</em></a> is <a class="reference
internal" href="#term-shutdown"><em >shutdown</em></a>
or <a href="#term-halted"><em class="xref std
std-term">halted</em></a> the job <a
href="#term-scheduling"><em >scheduling</em></a> is
suspended.
If this suspension is left for period of time, then it can affect task
submission under <strong>hybrid</strong> and <strong>real</strong> clocks.
In particular it will affect <a
href="#term-task"><em >task</em></a> s with <a
href="#term-time"><em class="xref std
std-term">time</em></a>, <a href="#term-today"><em
>today</em></a> or <a
href="#term-cron"><em >cron</em></a> <a
href="#term-dependencies"><em class="xref std
std-term">dependencies</em></a>.</p>
<blockquote>
<div><ul>
<li><p ><a
href="#term-dependencies"><em >dependencies</em></a>
with time series, can result in missed time slots:</p>
<div ><pre>time 10:00 20:00 00:15 # If server is
suspended > 15 minutes, time slots can be missed
time +00:05 20:00 00:15 # start 5 minutes after the start of the suite, then
every 15m until 20:00</pre>
</div>
</li>
<li><p >When the server is placed back into <a class="reference
internal" href="#term-running"><em >running</em></a>
state any time <a href="#term-dependencies"><em
>dependencies</em></a>
with an expired time slot are submitted straight away. i.e if <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> is
<a href="#term-halted"><em class="xref std
std-term">halted</em></a> at 10:59 and then placed back into <a
href="#term-running"><em class="xref std
std-term">running</em></a> state at 11:20</p>
<blockquote>
<div><p>time 11:00</p>
</div></blockquote>
<p>Then any <a href="#term-task"><em class="xref std
std-term">task</em></a> with a expired single time slot dependency will be
submitted straight away.</p>
</li>
</ul>
</div></blockquote>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Clock"
title="ecflow.Clock"><tt ><span
>ecflow.Clock</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Suite.add_clock"
title="ecflow.Suite.add_clock"><tt class="xref py py-class docutils
literal"><span >ecflow.Suite.add_clock</span></tt></a>. For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-clock"><tt
><span
>clock</span></tt></a></p>
</dd>
<dt id="term-complete">complete</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> <a
href="#term-status"><em >status</em></a>.</p>
<p>The node can be set to complete:</p>
<blockquote >
<div><p>By the <a
href="#term-complete-expression"><em >complete
expression</em></a></p>
<p>At job end when the <a href="#term-task"><em
>task</em></a> receives the <a class="reference
internal" href="#term-ecflow-client"><em class="xref std
std-term">ecflow_client</em></a> –complete <a
href="#term-child-command"><em >child
command</em></a></p>
</div></blockquote>
</dd>
<dt id="term-complete-expression">complete expression</dt>
<dd><p >Force a node to be complete <strong>if</strong> the
expression evaluates, without running any of the nodes.</p>
<p>This allows you to have tasks in the suite which a run only if others fail.
In practice the node would need to have a <a
href="#term-trigger"><em >trigger</em></a> also.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Expression"
title="ecflow.Expression"><tt ><span
>ecflow.Expression</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_complete"
title="ecflow.Node.add_complete"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_complete</span></tt></a></p>
</dd>
<dt id="term-cron">cron</dt>
<dd><p >Like <a href="#term-time"><em
>time</em></a>, cron defines time dependency for a <a
href="#term-node"><em class="xref std
std-term">node</em></a>, but it will be repeated indefinitely</p>
<div ><pre>cron 11:00
cron 10:00 22:00 00:30 # <start> <finish> <increment></pre>
</div>
<p>When the node becomes complete it will be <a
href="#term-queued"><em >queued</em></a> immediately.
This means that the suite
will never complete, and the output is not directly accessible through <a
href="#term-ecflowview"><em class="xref std
std-term">ecflowview</em></a></p>
<p>If tasks abort, the <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
will not schedule it again.</p>
<p>If the time the job takes to complete is longer than the interval a “slot”
is missed,
e.g.</p>
<div ><pre>cron 10:00 20:00 01:00</pre>
</div>
<p>if the 10:00 run takes more than an hour, the 11:00 run will never
occur.</p>
<p>If the cron defines months, days of the month, or week days or a single time
slot
the it relies on a day change, hence if a <a
href="#term-hybrid-clock"><em >hybrid clock</em></a>
is defined,
then it will be set to <a href="#term-complete"><em
>complete</em></a> at the beginning of the <a
href="#term-suite"><em class="xref std
std-term">suite</em></a>,
without running the corresponding job.
Otherwise under a hybrid clock the <a
href="#term-suite"><em >suite</em></a> would never <a
href="#term-complete"><em class="xref std
std-term">complete</em></a>.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Cron"
title="ecflow.Cron"><tt ><span
>ecflow.Cron</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_cron"
title="ecflow.Node.add_cron"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_cron</span></tt></a>. For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-cron"><tt
><span
>cron</span></tt></a></p>
</dd>
<dt id="term-date">date</dt>
<dd><p >This defines a date dependency for a node.</p>
<p>There can be multiple date dependencies.
The European format is used for dates, which is: dd.mm.yy as in 31.12.2007.
Any of the three number fields can be expressed with a wildcard <cite>*</cite>
to mean any valid value.
Thus, 01.*.* means the first day of every month of every year.</p>
<p>If a <a href="#term-hybrid-clock"><em class="xref
std std-term">hybrid clock</em></a> is defined, any node held by a date
dependency will be set to <a
href="#term-complete"><em >complete</em></a> at the
beginning
of the <a href="#term-suite"><em class="xref std
std-term">suite</em></a>, without running the corresponding job. Otherwise
under a hybrid clock the <a href="#term-suite"><em
>suite</em></a> would
never <a href="#term-complete"><em class="xref std
std-term">complete</em></a>.</p>
<p >For python see: <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Date"
title="ecflow.Date"><tt ><span
>ecflow.Date</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_date"
title="ecflow.Node.add_date"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_date</span></tt></a>. For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-date"><tt
><span
>date</span></tt></a></p>
</dd>
<dt id="term-day">day</dt>
<dd><p >This defines a day dependency for a node.</p>
<p>There can be multiple day dependencies.</p>
<p>If a <a href="#term-hybrid-clock"><em class="xref
std std-term">hybrid clock</em></a> is defined, any node held by a day
dependency will be set to <a
href="#term-complete"><em >complete</em></a> at the
beginning
of the <a href="#term-suite"><em class="xref std
std-term">suite</em></a>, without running the corresponding job. Otherwise
under a hybrid clock the <a href="#term-suite"><em
>suite</em></a> would
never <a href="#term-complete"><em class="xref std
std-term">complete</em></a>.</p>
<p >For python see: <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Day" title="ecflow.Day"><tt
><span
>ecflow.Day</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_day"
title="ecflow.Node.add_day"><tt ><span
>ecflow.Node.add_day</span></tt></a>. For text BNF see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-day"><tt
><span
>day</span></tt></a></p>
</dd>
<dt id="term-defstatus">defstatus</dt>
<dd><p >Defines the default <a
href="#term-status"><em >status</em></a> for a
task/family to be assigned to the <a
href="#term-node"><em >node</em></a> when the begin
command is issued.</p>
<p>By default <a href="#term-node"><em class="xref
std std-term">node</em></a> gets queued when you use begin on a <a
href="#term-suite"><em class="xref std
std-term">suite</em></a>.
defstatus is useful in preventing suites from running automatically once begun
or in setting
tasks complete so they can be run selectively.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.DState"
title="ecflow.DState"><tt ><span
>ecflow.DState</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_defstatus"
title="ecflow.Node.add_defstatus"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_defstatus</span></tt></a>. For text
BNF see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-defstatus"><tt
><span
>defstatus</span></tt></a></p>
</dd>
<dt id="term-dependencies">dependencies</dt>
<dd><p >Dependencies are attributes of node, that can
suppress/hold a <a href="#term-task"><em class="xref
std std-term">task</em></a> from taking part in <a
href="#term-job-creation"><em >job
creation</em></a>.</p>
<p>They include <a href="#term-trigger"><em
>trigger</em></a>, <a
href="#term-date"><em >date</em></a>, <a
href="#term-day"><em class="xref std
std-term">day</em></a>, <a href="#term-time"><em
>time</em></a>, <a
href="#term-today"><em >today</em></a>, <a
href="#term-cron"><em class="xref std
std-term">cron</em></a>, <a
href="#term-complete-expression"><em >complete
expression</em></a>, <a href="#term-inlimit"><em
>inlimit</em></a> and <a
href="#term-limit"><em >limit</em></a>.</p>
<p>A <a href="#term-task"><em class="xref std
std-term">task</em></a> that is dependent can not be started as long as some
dependency is holding it or any of its <strong>parent</strong> <a
href="#term-node"><em class="xref std
std-term">node</em></a> s.</p>
<p >The <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
will check the dependencies every minute, during normal <a class="reference
internal" href="#term-scheduling"><em class="xref std
std-term">scheduling</em></a> <strong>and</strong> when any
<a href="#term-child-command"><em class="xref std
std-term">child command</em></a> causes a state change in the <a
href="#term-suite-definition"><em class="xref std
std-term">suite definition</em></a>.</p>
</dd>
<dt id="term-directives">directives</dt>
<dd><p >Directives start with a % charater. This is referred to as
<a href="#term-ecf-micro"><em class="xref std
std-term">ECF_MICRO</em></a> character.</p>
<p>The directives are used in two main context.</p>
<blockquote>
<div><ul>
<li><p >Preprocessing directives. In this case the directive
starts as the <strong>first</strong> character on a line in a <a
href="#term-ecf-script"><em class="xref std
std-term">ecf script</em></a> file.
See the table below which shows the allowable values. Only one directive is
allowed on the line.</p>
</li>
<li><p >Variable directives. We use two <a class="reference
internal" href="#term-ecf-micro"><em class="xref std
std-term">ECF_MICRO</em></a> characters ie %VAR%, in this case they can occur
<strong>anywhere</strong> on
the line and in any number.</p>
<div ><pre>%CAR% %TYPE% %WISHLIST%</pre>
</div>
<p>These directives take part in <a
href="#term-variable-substitution"><em >variable
substitution</em></a>.</p>
<p>If the micro characters are not paired (i.e uneven) then <a class="reference
internal" href="#term-variable-substitution"><em class="xref std
std-term">variable substitution</em></a> can not take place
hence an error message is issued.</p>
<div ><pre>port=%ECF_PORT # error issued since
'%' micro character are not paired.</pre>
</div>
<p>However an uneven number of micro character are allowed, <strong>If</strong>
the line begins with ‘#’ comment charcter.</p>
<div ><div ><pre><span >#
This is a comment line with a single micro character % no error issued</span>
<span ># port=%ECF_PORT again no error issued</span>
</pre></div>
</div>
</li>
</ul>
</div></blockquote>
<p>Directives are expanded during <a
href="#term-pre-processing"><em class="xref std
std-term">pre-processing</em></a>. Examples include:</p>
<table border="1" >
<colgroup>
<col width="22%" />
<col width="78%" />
</colgroup>
<thead valign="bottom">
<tr ><th >Symbol</th>
<th >Meaning</th>
</tr>
</thead>
<tbody valign="top">
<tr ><td>%include <filename></td>
<td>%ECF_INCLUDE% directory is searched for the <tt class="file docutils
literal"><span >filename</span></tt> and the contents
included into the job file. If that variable is not defined ECF_HOME is used.
If the ECF_INCLUDE is defined but the file does not exist, then we look in
ECF_HOME. This allows specific files to be placed in ECF_INCLUDE and the
more general/common include files to be placed in ECF_HOME.
This is the recommended format</td>
</tr>
<tr ><td>%include “filename”</td>
<td>Include the contents of the file:
%ECF_HOME%/%SUITE%/%FAMILY%/filename into the job.</td>
</tr>
<tr ><td>%include filename</td>
<td>Include the contents of the file <tt ><span
>filename</span></tt> into the output. The only form
that can be used safely must start with a slash ‘/’</td>
</tr>
<tr ><td>%includenopp filename</td>
<td>Same as %include, but the file is not interpreted at all.</td>
</tr>
<tr ><td>%comment</td>
<td>Start’s a comment, which is ended by %end directive.
The section enclosed by %comment - %end is removed during <a class="reference
internal" href="#term-pre-processing"><em class="xref std
std-term">pre-processing</em></a></td>
</tr>
<tr ><td>%manual</td>
<td>Start’s a manual, which is ended by %end directive.
The section enclosed by %manual - %end is removed during <a class="reference
internal" href="#term-pre-processing"><em class="xref std
std-term">pre-processing</em></a>.
The manual directive is used to create the <a
href="#term-manual-page"><em >manual page</em></a>
show in <a href="#term-ecflowview"><em class="xref
std std-term">ecflowview</em></a>.</td>
</tr>
<tr ><td>%nopp</td>
<td>Stop pre-processing until a line starting with %end is found.
No interpretation of the text will be done( i.e. no variable
substitutions)</td>
</tr>
<tr ><td>%end</td>
<td>End processing of %comment or %manual or %nopp</td>
</tr>
<tr ><td>%ecfmicro CHAR</td>
<td>Change the directive character, to the character given. If set in an
include file the effect is retained for the rest of the job( or until
set again). It should be noted that the ecfmicro directive specified in
the <a href="#term-ecf-script"><em class="xref std
std-term">ecf script</em></a> file, does <strong>not</strong> effect the
variable substitution
for ECF_JOB_CMD, ECF_KILL_CMD or ECF_STATUS_CMD variables. They still use
<a href="#term-ecf-micro"><em class="xref std
std-term">ECF_MICRO</em></a>. If no ecfmicro directive exists, we default to
using
<a href="#term-ecf-micro"><em class="xref std
std-term">ECF_MICRO</em></a> from the <a
href="#term-suite-definition"><em >suite
definition</em></a></td>
</tr>
</tbody>
</table>
</dd>
<dt id="term-ecf-file-location-algorithm">ecf file location algorithm</dt>
<dd><p ><a
href="#term-ecflow-server"><em >ecflow_server</em></a>
and job creation checking uses the following algorithm to locate the
‘.ecf’ file corresponding to a <a
href="#term-task"><em >task</em></a>:</p>
<ul>
<li><p >ECF_SCRIPT</p>
<p>First it uses the generated variable ECF_SCRIPT to locate the script.
This variable is generated from: ECF_HOME/<path to task>.ecf</p>
<p>Hence if the task path is /suite/f1/f2/t1, then
ECF_SCRIPT=ECF_HOME/suite/f1/f2/t1.ecf</p>
</li>
<li><p >ECF_FILES</p>
<p>Second it checks for the user defined ECF_FILES variable.
If defined the value of this variable must correspond to a directory.
This directory is searched in reverse order.</p>
<p>i.e lets assume we have a task: /o/12/fc/model
and ECF_FILES is defined as: /home/ecmwf/emos/def/o/ECFfiles</p>
<p>The ecFlow will use the following search pattern.</p>
<blockquote>
<div><ol >
<li>/home/ecmwf/emos/def/o/ECFfiles/o/12/fc/model.ecf</li>
<li>/home/ecmwf/emos/def/o/ECFfiles/12/fc/model.ecf</li>
<li>/home/ecmwf/emos/def/o/ECFfiles/fc/model.ecf</li>
<li>/home/ecmwf/emos/def/o/ECFfiles/model.ecf</li>
</ol>
</div></blockquote>
</li>
<li><p >ECF_HOME</p>
<p>Thirdly it searchs for the script in reverse order using ECF_HOME (i.e like
ECF_FILES)</p>
</li>
</ul>
<blockquote >
<div><p>If this fails, than the <a
href="#term-task"><em >task</em></a> is placed into
the <a href="#term-aborted"><em class="xref std
std-term">aborted</em></a> state.
We can check that file can be located before loading the suites into the
server.</p>
<blockquote>
<div><ul >
<li><a
href="/wiki/display/ECFLOW/Checking+job+creation#checking-job-creation"><em>Checking
job creation</em></a></li>
<li><a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Defs.check_job_creation"
title="ecflow.Defs.check_job_creation"><tt class="xref py py-class docutils
literal"><span >ecflow.Defs.check_job_creation</span></tt></a></li>
</ul>
</div></blockquote>
</div></blockquote>
</dd>
<dt id="term-ecf-script">ecf script</dt>
<dd><p >The ecFlow script refers to an ‘.ecf’ file.</p>
<p>The script file is transformed into the <a
href="#term-job-file"><em >job file</em></a> by the <a
href="#term-job-creation"><em class="xref std
std-term">job creation</em></a> process.</p>
<p>The base name of the script file <strong>must</strong> match its
corresponding <a href="#term-task"><em class="xref
std std-term">task</em></a>. i.e t1.ecf , corresponds to the task of name
‘t1’.
The script if placed in the ECF_FILES directory, may be re-used by multiple
tasks belonging to different families,
providing the <a href="#term-task"><em class="xref
std std-term">task</em></a> name matches.</p>
<p>The ecFlow script is similar to a UNIX shell script.</p>
<p >The differences, however, includes the addition of “C” like
pre-processing <a href="#term-directives"><em
>directives</em></a> and ecFlow <a class="reference
internal" href="#term-variable"><em class="xref std
std-term">variable</em></a>‘s.
Also the script <em>must</em> include calls to the <strong>init</strong> and
<strong>complete</strong> <a
href="#term-child-command"><em >child command</em></a>
s so that
the <a href="#term-ecflow-server"><em class="xref
std std-term">ecflow_server</em></a> is aware when the job starts(i.e changes
state to <a href="#term-active"><em class="xref std
std-term">active</em></a>) and finishes ( i.e changes state to <a
href="#term-complete"><em class="xref std
std-term">complete</em></a>)</p>
</dd>
<dt id="term-ecf-dummy-task">ECF_DUMMY_TASK</dt>
<dd><p >This is a user variable that can be added to <a
href="#term-task"><em class="xref std
std-term">task</em></a> to indicate that there is no
associated <a href="#term-ecf-script"><em
>ecf script</em></a> file.</p>
<p>If this variable is added to <a
href="#term-suite"><em >suite</em></a> or <a
href="#term-family"><em class="xref std
std-term">family</em></a> then all child tasks are treated as dummy.</p>
<p >This stops the server from reporting an error during <a
href="#term-job-creation"><em class="xref std
std-term">job creation</em></a>.</p>
</dd>
<dt id="term-ecf-job">ECF_JOB</dt>
<dd><p >This is a generated <a
href="#term-variable"><em >variable</em></a>. If
defines the path name location of the job file.</p>
<p >The variable is composed as:
ECF_HOME/ECF_NAME.job<ECF_TRYNO></p>
</dd>
<dt id="term-ecf-jobout">ECF_JOBOUT</dt>
<dd><p >This is a generated <a
href="#term-variable"><em >variable</em></a>. This
variable defines the path name
for the job output file. The variable is composed as:</p>
<blockquote >
<div>ECF_HOME/ECF_NAME.ECF_TRYNO</div></blockquote>
</dd>
<dt id="term-ecf-micro">ECF_MICRO</dt>
<dd>This is a generated <a href="#term-variable"><em
>variable</em></a>. The default value is %.
This variable is used in <a
href="#term-variable-substitution"><em >variable
substitution</em></a> during command invocation and
default directive character during <a
href="#term-pre-processing"><em class="xref std
std-term">pre-processing</em></a>.
It can be overriden, but must be replaced by a single character.</dd>
<dt id="term-ecf-name">ECF_NAME</dt>
<dd>This is a generated <a href="#term-variable"><em
>variable</em></a>. It defines the path name of the
task.</dd>
<dt id="term-63">ECF_SCRIPT</dt>
<dd>This is a generated <a href="#term-variable"><em
>variable</em></a>. If defines the path name for the
<a href="#term-ecf-script"><em class="xref std
std-term">ecf script</em></a></dd>
<dt id="term-ecf-tryno">ECF_TRYNO</dt>
<dd><p >This is a generated <a
href="#term-variable"><em >variable</em></a> that is
used in file name generation.
It represents the current try number for the <a
href="#term-task"><em >task</em></a>.</p>
<p >After <strong>begin</strong> it is set to 1. The number is
advanced if the job is re-run.
It is re-set back to 1 after a re-queue.
It is used in output and <a
href="#term-job-file"><em >job file</em></a>
numbering.
(i.e It avoids overwriting the <a
href="#term-job-file"><em >job file</em></a> output
during multiple re-runs)</p>
</dd>
<dt id="term-ecflow">ecFlow</dt>
<dd><p >Is the ECMWF work flow manager.</p>
<p>A general purpose application designed to schedule a large number of
computer process in a heterogeneous environment.</p>
<p >Helps computer jobs design, submission and monitoring both in
the research and operation departments.</p>
</dd>
<dt id="term-ecflow-client">ecflow_client</dt>
<dd><p >This executable is a command line program; it is used for
all communication with the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>.</p>
<p>To see the full range of commands that can be sent to the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> type the following in a UNIX shell:</p>
<blockquote>
<div>ecflow_client –help</div></blockquote>
<p>This functionality is also provided by the <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#client-server-python-api"><em>Client
Server API</em></a>.</p>
<p>The following variables affect the execution of ecflow_client.</p>
<p>Since the <a href="#term-ecf-script"><em
>ecf script</em></a> can call ecflow_client( i.e <a
href="#term-child-command"><em class="xref std
std-term">child command</em></a>) then typically
some are set in an include header. i.e. <a
href="/wiki/display/ECFLOW/Understanding+Includes#head-h"><em>head.h</em></a>.</p>
<table border="1" >
<colgroup>
<col width="12%" />
<col width="49%" />
<col width="8%" />
<col width="31%" />
</colgroup>
<thead valign="bottom">
<tr ><th >Variable Name</th>
<th >Explanation</th>
<th >Compulsory</th>
<th >Example</th>
</tr>
</thead>
<tbody valign="top">
<tr ><td>ECF_NODE</td>
<td>Name of the host running the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></td>
<td>Yes</td>
<td>pikachu</td>
</tr>
<tr ><td>ECF_NAME</td>
<td>Path to the task</td>
<td>Yes</td>
<td>/suite/family/task</td>
</tr>
<tr ><td>ECF_PASS</td>
<td>Jobs password.</td>
<td>Yes</td>
<td>(generated)</td>
</tr>
<tr ><td>ECF_RID</td>
<td>Remote id. Allow easier job kill, and disambiguate a zombie
from the real job.</td>
<td>Yes</td>
<td>(generated)</td>
</tr>
<tr ><td>ECF_PORT</td>
<td>Port number of the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></td>
<td>No</td>
<td>3141.Must match <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></td>
</tr>
<tr ><td>ECF_TRYNO</td>
<td>The number of times the job has run. This is allocated by
the server and used in job/output file name generation.</td>
<td>No</td>
<td>(generated)</td>
</tr>
<tr ><td>ECF_HOSTFILE</td>
<td>File that lists alternate hosts to try, if connection to
main host fails</td>
<td>No</td>
<td>/home/user/avi/.ecfhostfile</td>
</tr>
<tr ><td>ECF_TIMEOUT</td>
<td>Maximum time is seconds for the client to deliver message</td>
<td>No</td>
<td>24*3600 (default value)</td>
</tr>
<tr ><td>ECF_DENIED</td>
<td>If server denies client communication and this flag is set,
exit with an error. Avoids 24hr hour connection attempt to
<a href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>.</td>
<td>No</td>
<td>1</td>
</tr>
<tr ><td>NO_ECF</td>
<td>If set exit’s ecflow_client immediately with success. This
allows the scripts to be tested independent of the server</td>
<td>No</td>
<td>1</td>
</tr>
</tbody>
</table>
</dd>
<dt id="term-ecflow-server">ecflow_server</dt>
<dd><p >This executable is the server.</p>
<p>It is responsible for <a
href="#term-scheduling"><em >scheduling</em></a> the
jobs and responding to <a
href="#term-ecflow-client"><em >ecflow_client</em></a>
requests</p>
<p>Multiple servers can be run on the same machine/host providing they are
assigned a unique port number.</p>
<p>The server record’s all request’s in the log file.</p>
<p>The server will periodically(See ECF_CHECKINTERVAL) write out a <a
href="#term-check-point"><em class="xref std
std-term">check point</em></a> file.</p>
<p>The following environment variables control the execution of the server and
may be set before the start of the server.
ecflow_server will start happily with out any of these variables being set,
since all of them have default values.</p>
<table border="1" >
<colgroup>
<col width="15%" />
<col width="51%" />
<col width="34%" />
</colgroup>
<thead valign="bottom">
<tr ><th >Variable Name</th>
<th >Explanation</th>
<th >Default value</th>
</tr>
</thead>
<tbody valign="top">
<tr ><td>ECF_HOME</td>
<td>Home for all the <a href="#term-ecflow"><em
>ecFlow</em></a> files</td>
<td>Current working directory</td>
</tr>
<tr ><td>ECF_PORT</td>
<td>Server port number. Must be unique</td>
<td>3141</td>
</tr>
<tr ><td>ECF_LOG</td>
<td>History or log file</td>
<td><host>.<port>.ecf.log</td>
</tr>
<tr ><td>ECF_CHECK</td>
<td>Name of the checkpoint file</td>
<td><host>.<port>.ecf.check</td>
</tr>
<tr ><td>ECF_CHECKOLD</td>
<td>Name of the backup checkpoint file</td>
<td><host>.<port>.ecf.check.b</td>
</tr>
<tr ><td>ECF_CHECKINTERVAL</td>
<td>Interval in second to save <a
href="#term-check-point"><em >check point</em></a>
file</td>
<td>120</td>
</tr>
<tr ><td>ECF_LISTS</td>
<td>White list file. Controls read/write access to the server
for each user</td>
<td><host>.<port>.ecf.lists</td>
</tr>
</tbody>
</table>
<p >The server can be in several states. The default when first
started is <a href="#term-halted"><em class="xref
std std-term">halted</em></a>, See <a
href="#term-server-states"><em >server
states</em></a></p>
</dd>
<dt id="term-ecflowview">ecflowview</dt>
<dd><p >ecflowview executable is the GUI based client, that is
used to visualise and monitor
the hierarchical structure of the <a
href="#term-suite-definition"><em >suite
definition</em></a></p>
<blockquote >
<div><p>state changes in the <a
href="#term-node"><em >node</em></a>‘s and the
<a href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>, using colour coding</p>
<p>Attributes of the nodes and any <a
href="#term-dependencies"><em class="xref std
std-term">dependencies</em></a></p>
<p><a href="#term-ecf-script"><em class="xref std
std-term">ecf script</em></a> file and the corresponding <a class="reference
internal" href="#term-job-file"><em >job
file</em></a></p>
</div></blockquote>
</dd>
<dt id="term-event">event</dt>
<dd><p >The purpose of an event is to signal partial completion of
a <a href="#term-task"><em class="xref std
std-term">task</em></a> and to be able to
trigger another job which is waiting for this partial completion.</p>
<p>Only tasks can have events and they can be considered as an attribute of a
<a href="#term-task"><em class="xref std
std-term">task</em></a>.</p>
<p>There can be many events and they are displayed as nodes.</p>
<p>The event is updated by placing the –event <a class="reference
internal" href="#term-child-command"><em >child
command</em></a> in a <a href="#term-ecf-script"><em
>ecf script</em></a>.</p>
<p>An event has a number and possibly a name. If it is only defined as a
number,
its name is the text representation of the number without leading zeroes.</p>
<p>For python see: <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Event"
title="ecflow.Event"><tt ><span
>ecflow.Event</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_event"
title="ecflow.Node.add_event"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_event</span></tt></a> For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-event"><tt
><span
>event</span></tt></a></p>
<p >Events can be referenced in <a
href="#term-trigger"><em >trigger</em></a> and <a
href="#term-complete-expression"><em class="xref std
std-term">complete expression</em></a> s.</p>
</dd>
<dt id="term-extern">extern</dt>
<dd><p >This allows an external <a
href="#term-node"><em >node</em></a> to be used in a
<a href="#term-trigger"><em class="xref std
std-term">trigger</em></a> expression.</p>
<p>All <a href="#term-node"><em class="xref std
std-term">node</em></a>‘s in <a
href="#term-trigger"><em >trigger</em></a>‘s
must be known to <a href="#term-ecflow-server"><em
>ecflow_server</em></a> by the end of the load
command.
No cross-suite <a href="#term-dependencies"><em
>dependencies</em></a> are allowed unless the names of
tasks outside the suite are declared as external.
An external <a href="#term-trigger"><em class="xref
std std-term">trigger</em></a> reference is considered unknown if it is not
defined when the <a href="#term-trigger"><em
>trigger</em></a> is evaluated.
You are strongly advised to avoid cross-suite <a
href="#term-dependencies"><em class="xref std
std-term">dependencies</em></a>.</p>
<p >Families and suites that depend on one another should be placed
in a single <a href="#term-suite"><em class="xref
std std-term">suite</em></a>.
If you think you need cross-suite dependencies, you should consider merging the
suites
together and have each as a top-level family in the merged suite.
For BNF see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-extern"><tt
><span
>extern</span></tt></a></p>
</dd>
<dt id="term-family">family</dt>
<dd><p >A family is an organisational entity that is used to
provide hierarchy and grouping.
It consists of a collection of <a
href="#term-task"><em >task</em></a>‘s and
families.</p>
<p>Typically you place tasks that are related to each other inside the same
family, analogous to the way you
create directories to contain related files.
For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Family"
title="ecflow.Family"><tt ><span
>ecflow.Family</span></tt></a>. For BNF see <a class="reference
internal"
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-family"><tt
><span
>family</span></tt></a></p>
<p >It serves as an intermediate <a
href="#term-node"><em >node</em></a> in a <a
href="#term-suite-definition"><em class="xref std
std-term">suite definition</em></a>.</p>
</dd>
<dt id="term-halted">halted</dt>
<dd>Is a <a href="#term-ecflow-server"><em
>ecflow_server</em></a> state. See <a class="reference
internal" href="#term-server-states"><em >server
states</em></a></dd>
<dt id="term-hybrid-clock">hybrid clock</dt>
<dd><p >A hybrid <a
href="#term-clock"><em >clock</em></a> is a complex
notion: the date and time are not connected.</p>
<p>The date has a fixed value during the complete execution of the <a
href="#term-suite"><em class="xref std
std-term">suite</em></a>.
This will be mainly used in cases where the suite does not <a class="reference
internal" href="#term-complete"><em >complete</em></a>
in less than 24 hours.
This guarantees that all tasks of this suite are using the same <a
href="#term-date"><em class="xref std
std-term">date</em></a>.
On the other hand, the time follows the time of the machine.</p>
<p>Hence the <a href="#term-date"><em class="xref
std std-term">date</em></a> never changes unless specifically altered or unless
the suite restarts,
either automatically or from a begin command.</p>
<p >Under a hybrid <a
href="#term-clock"><em >clock</em></a> any <a
href="#term-node"><em class="xref std
std-term">node</em></a> held by a <a
href="#term-date"><em >date</em></a> or <a
href="#term-day"><em class="xref std
std-term">day</em></a> dependency
will be set to complete at the beginning of the suite. (i.e without its job
ever running).
Otherwise the <a href="#term-suite"><em class="xref
std std-term">suite</em></a> would never <a
href="#term-complete"><em >complete</em></a>.</p>
</dd>
<dt id="term-inlimit">inlimit</dt>
<dd><p >The inlimit works in conjunction with <a class="reference
internal" href="#term-limit"><em >limit</em></a>/<a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Limit"
title="ecflow.Limit"><tt ><span
>ecflow.Limit</span></tt></a> for providing simple load
management</p>
<p>inlimit is added to the <a href="#term-node"><em
>node</em></a> that needs to be limited.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.InLimit"
title="ecflow.InLimit"><tt ><span
>ecflow.InLimit</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_inlimit"
title="ecflow.Node.add_inlimit"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_inlimit</span></tt></a>. For text
BNF see <a href="#term-inlimit"><em class="xref std
std-term">inlimit</em></a></p>
</dd>
<dt id="term-job-creation">job creation</dt>
<dd><p >Job creation or task invocation can be initiated manually
via <a href="#term-ecflowview"><em class="xref std
std-term">ecflowview</em></a> but also by
the <a href="#term-ecflow-server"><em class="xref
std std-term">ecflow_server</em></a> during <a
href="#term-scheduling"><em >scheduling</em></a> when
a <a href="#term-task"><em class="xref std
std-term">task</em></a> (and <em>all</em> of its parent <a class="reference
internal" href="#term-node"><em >node</em></a> s)
is free of its <a href="#term-dependencies"><em
>dependencies</em></a>.</p>
<p>The process of job creation includes:</p>
<blockquote>
<div><p>o Generating a unique password ECF_PASS, which is placed in <a
href="#term-ecf-script"><em class="xref std
std-term">ecf script</em></a> during <a
href="#term-pre-processing"><em class="xref std
std-term">pre-processing</em></a>. See <a
href="/wiki/display/ECFLOW/Understanding+Includes#head-h"><em>head.h</em></a></p>
<p>o Locating <a href="#term-ecf-script"><em
>ecf script</em></a> files , corresponding to the <a
href="#term-task"><em class="xref std
std-term">task</em></a> in the <a
href="#term-suite-definition"><em >suite
definition</em></a>, See <a
href="#term-ecf-file-location-algorithm"><em >ecf file
location algorithm</em></a></p>
<p>o <a href="#term-pre-processing"><em class="xref
std std-term">pre-processing</em></a> the contents of the <a class="reference
internal" href="#term-ecf-script"><em >ecf
script</em></a> file</p>
</div></blockquote>
<p>The steps above transforms an <a
href="#term-ecf-script"><em >ecf script</em></a> to a
<a href="#term-job-file"><em class="xref std
std-term">job file</em></a> that can be submitted by
performing <a href="#term-variable-substitution"><em
>variable substitution</em></a> on the ECF_JOB_CMD <a
href="#term-variable"><em class="xref std
std-term">variable</em></a> and invoking the command.</p>
<p>The running jobs will communicate back to the <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
by calling <a href="#term-child-command"><em
>child command</em></a>‘s.</p>
<p>This causes <a href="#term-status"><em
>status</em></a> changes on the <a class="reference
internal" href="#term-node"><em >node</em></a>‘s
in the <a href="#term-ecflow-server"><em class="xref
std std-term">ecflow_server</em></a> and flags can be set
to indicate various events.</p>
<p>If a <a href="#term-task"><em class="xref std
std-term">task</em></a> is to be treated as a dummy task( i.e. is used as a
scheduling task) and is not meant to
to be run, then a variable of name <a
href="#term-ecf-dummy-task"><em class="xref std
std-term">ECF_DUMMY_TASK</em></a> can be added.</p>
<div ><div ><pre><span
>task</span><span >.</span><span
>add_variable</span><span >(</span><span
>"ECF_DUMMY_TASK"</span><span >,</span><span
>""</span><span >)</span>
</pre></div>
</div>
</dd>
<dt id="term-job-file">job file</dt>
<dd><p >The job file is created by the <a class="reference
internal" href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> during <a
href="#term-job-creation"><em >job creation</em></a>
using the <a href="#term-ecf-tryno"><em class="xref
std std-term">ECF_TRYNO</em></a> <a
href="#term-variable"><em >variable</em></a></p>
<p>It is derived from the <a
href="#term-ecf-script"><em >ecf script</em></a> after
expanding the pre-processing <a
href="#term-directives"><em >directives</em></a>.</p>
<p>It has the form <task name>.job<<a
href="#term-ecf-tryno"><em class="xref std
std-term">ECF_TRYNO</em></a>>”, i.e. t1.job1.</p>
<p>Note job creation checking will create a job file with an extension with
zero. i.e ‘.job0’. See <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Defs.check_job_creation"
title="ecflow.Defs.check_job_creation"><tt class="xref py py-class docutils
literal"><span >ecflow.Defs.check_job_creation</span></tt></a></p>
<p >When the job is run the output file has the <a class="reference
internal" href="#term-ecf-tryno"><em class="xref std
std-term">ECF_TRYNO</em></a> as the extension.
i.e t1.1 where ‘t1’ represents the task name and ‘1’
the <a href="#term-ecf-tryno"><em class="xref std
std-term">ECF_TRYNO</em></a></p>
</dd>
<dt id="term-label">label</dt>
<dd><p >A label has a name and a value and is a way of
<strong>displaying</strong> information in <a
href="#term-ecflowview"><em >ecflowview</em></a></p>
<p>By placing a label <a
href="#term-child-command"><em >child command</em></a>
s in the <a href="#term-ecf-script"><em class="xref
std std-term">ecf script</em></a> the user can be informed about progress
in <a href="#term-ecflowview"><em class="xref std
std-term">ecflowview</em></a>.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Label"
title="ecflow.Label"><tt ><span
>ecflow.Label</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_label"
title="ecflow.Node.add_label"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_label</span></tt></a>. For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-label"><tt
><span
>label</span></tt></a></p>
</dd>
<dt id="term-late">late</dt>
<dd><p >Define a tag for a node to be late.</p>
<p>Suites cannot be late, but you can define a late tag
for submitted in a suite, to be inherited by the families and tasks.
When a node is classified as being late, the only action <a class="reference
internal" href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> takes is to set a flag.
<a href="#term-ecflowview"><em class="xref std
std-term">ecflowview</em></a> will display these alongside the <a
href="#term-node"><em class="xref std
std-term">node</em></a> name as an icon (and optionally pop up a window).</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Late"
title="ecflow.Late"><tt ><span
>ecflow.Late</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_late"
title="ecflow.Node.add_late"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_late</span></tt></a>. For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-late"><tt
><span
>late</span></tt></a></p>
</dd>
<dt id="term-limit">limit</dt>
<dd><p >Limits provide simple load management by limiting the
number of tasks
submitted by a specific <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>.
Typically you either define limits on <a
href="#term-suite"><em >suite</em></a> level or define
a separate suite to
hold limits so that they can be used by multiple suites.</p>
<p>The limit max value can be changed on the command line</p>
<div ><pre>>ecflow_client --alter change limit_max
<limit-name> <new-limit-value> <path-to-limit>
>ecflow_client --alter change limit_max limit 2 /suite</pre>
</div>
<p>It can also be changed in python</p>
<div ><div ><pre><span
>#!/usr/bin/env python2.7</span>
<span >import</span> <span >ecflow</span>
<span >try</span><span >:</span>
<span >ci</span> <span >=</span> <span
>ecflow</span><span >.</span><span
>Client</span><span >()</span>
<span >ci</span><span >.</span><span
>alter</span><span >(</span><span
>"/suite"</span><span >,</span><span
>"change"</span><span >,</span><span
>"limit_max"</span><span >,</span><span
>"limit"</span><span >,</span> <span
>"2"</span><span >)</span>
<span >except</span> <span >RuntimeError</span><span
>,</span> <span >e</span><span >:</span>
<span >print</span> <span >"Failed: "</span>
<span >+</span> <span >str</span><span
>(</span><span >e</span><span >)</span>
</pre></div>
</div>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Limit"
title="ecflow.Limit"><tt ><span
>ecflow.Limit</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_limit"
title="ecflow.Node.add_limit"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_limit</span></tt></a>. For BNF see
<a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-limit"><tt
><span
>limit</span></tt></a> and <a
href="#term-inlimit"><em >inlimit</em></a></p>
</dd>
<dt id="term-manual-page">manual page</dt>
<dd><p >Manual pages are part of the <a
href="#term-ecf-script"><em >ecf script</em></a>.</p>
<p>This is to ensure that the manual page is updated when the <a
href="#term-ecf-script"><em class="xref std
std-term">ecf script</em></a> is updated.
The manual page is a very important operational tool allowing you to view a
description of a task,
and possibly describing solutions to common problems.
The <a href="#term-pre-processing"><em class="xref
std std-term">pre-processing</em></a> can be used to extract the manual page
from the script file and is visible in <a
href="#term-ecflowview"><em >ecflowview</em></a>.
The manual page is the text contained within the %manual and %end <a
href="#term-directives"><em class="xref std
std-term">directives</em></a>.
They can be seen using the manual button on <a
href="#term-ecflowview"><em >ecflowview</em></a>.</p>
<p>The text in the manual page in <strong>not</strong> included in the <a
href="#term-job-file"><em class="xref std
std-term">job file</em></a>.</p>
<p>There can be multiple manual sections in the same <a class="reference
internal" href="#term-ecf-script"><em >ecf
script</em></a> file. When viewed they are
simply concatenated. It is good practice to modify the manual pages when the
script changes.</p>
<p >The manual page may have the %include <a class="reference
internal" href="#term-directives"><em class="xref std
std-term">directives</em></a>.</p>
</dd>
<dt id="term-meter">meter</dt>
<dd><p >The purpose of a meter is to signal proportional
completion of a task and to
be able to trigger another job which is waiting on this proportional
completion.</p>
<p>The meter is updated by placing the –meter <a class="reference
internal" href="#term-child-command"><em >child
command</em></a> in a <a href="#term-ecf-script"><em
>ecf script</em></a>.</p>
<p>For python see: <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Meter"
title="ecflow.Meter"><tt ><span
>ecflow.Meter</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_meter"
title="ecflow.Node.add_meter"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_meter</span></tt></a>. For text BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-meter"><tt
><span
>meter</span></tt></a></p>
<p >Meter’s can be referenced in <a class="reference
internal" href="#term-trigger"><em >trigger</em></a>
and <a href="#term-complete-expression"><em
>complete expression</em></a> expressions.</p>
</dd>
<dt id="term-node">node</dt>
<dd><p ><a href="#term-suite"><em
>suite</em></a>, <a
href="#term-family"><em >family</em></a> and <a
href="#term-task"><em class="xref std
std-term">task</em></a> form a hierarchy.
Where a <a href="#term-suite"><em class="xref std
std-term">suite</em></a> serves as the root of the hierarchy.
The <a href="#term-family"><em class="xref std
std-term">family</em></a> provides the intermediate nodes, and the <a
href="#term-task"><em class="xref std
std-term">task</em></a>
provide the leaf’s.</p>
<p>Collectively <a href="#term-suite"><em
>suite</em></a>, <a
href="#term-family"><em >family</em></a> and <a
href="#term-task"><em class="xref std
std-term">task</em></a> can be referred
to as nodes.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node"
title="ecflow.Node"><tt ><span
>ecflow.Node</span></tt></a>.</p>
</dd>
<dt id="term-pre-processing">pre-processing</dt>
<dd><p >Pre-processing takes place during <a class="reference
internal" href="#term-job-creation"><em >job
creation</em></a> and acts on <a
href="#term-directives"><em >directives</em></a>
specified in <a href="#term-ecf-script"><em
>ecf script</em></a> file.</p>
<p>This involves:</p>
<blockquote >
<div><p>o expanding any include file <a
href="#term-directives"><em >directives</em></a>. i.e
similar to ‘c’ language pre-processing</p>
<p>o removing comments and manual <a
href="#term-directives"><em >directives</em></a></p>
<p>o performing <a
href="#term-variable-substitution"><em >variable
substitution</em></a></p>
</div></blockquote>
</dd>
<dt id="term-queued">queued</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> <a
href="#term-status"><em >status</em></a>.</p>
<p >After the begin command, the task <strong>without</strong> a <a
href="#term-defstatus"><em class="xref std
std-term">defstatus</em></a> are placed into the queued state</p>
</dd>
<dt id="term-real-clock">real clock</dt>
<dd>A <a href="#term-suite"><em class="xref std
std-term">suite</em></a> using a real <a
href="#term-clock"><em >clock</em></a> will have its
<a href="#term-clock"><em class="xref std
std-term">clock</em></a> matching the clock of the machine.
Hence the <a href="#term-date"><em class="xref std
std-term">date</em></a> advances by one day at midnight.</dd>
<dt id="term-repeat">repeat</dt>
<dd><p >Repeats provide looping functionality. There can only be a
single repeat on a <a href="#term-node"><em
>node</em></a>.</p>
<blockquote>
<div><p>repeat day step [ENDDATE] # only for suites</p>
<p>repeat integer VARIABLE start end [step]</p>
<p>repeat enumerated VARIABLE first [second [third ...]]</p>
<p>repeat string VARIABLE str1 [str2 ...]</p>
<p>repeat file VARIABLE filename</p>
<p>repeat date VARIABLE yyyymmdd yyyymmdd [delta]</p>
</div></blockquote>
<p>The repeat VARIABLE can be used in <a
href="#term-trigger"><em >trigger</em></a> and <a
href="#term-complete-expression"><em class="xref std
std-term">complete expression</em></a> expressions.</p>
<p>If a “repeat date” VARIABLE is used in a trigger expression then
date arithmetic is used,
when the expression uses addition and subtraction. i.e</p>
<div ><div ><pre><span
>defs</span> <span >=</span> <span
>ecflow</span><span >.</span><span >Defs</span><span
>()</span>
<span >s1</span> <span >=</span> <span
>defs</span><span >.</span><span
>add_suite</span><span >(</span><span
>"s1"</span><span >);</span>
<span >t1</span> <span >=</span> <span
>s1</span><span >.</span><span >add_task</span><span
>(</span><span >"t1"</span><span
>)</span><span >.</span><span
>add_repeat</span><span >(</span> <span
>ecflow</span><span >.</span><span
>RepeatDate</span><span >(</span><span
>"YMD"</span><span >,</span><span
>20090101</span><span >,</span><span
>20091231</span><span >,</span><span
>1</span><span >)</span> <span >);</span>
<span >t2</span> <span >=</span> <span
>s1</span><span >.</span><span >add_task</span><span
>(</span><span >"t2"</span><span
>)</span><span >.</span><span
>add_trigger</span><span >(</span><span
>"t1:YMD - 1 eq 20081231"</span><span >);</span>
<span >assert</span> <span >t2</span><span
>.</span><span >evaluate_trigger</span><span
>(),</span> <span >"Expected trigger to evaluate.
20090101 - 1 == 20081231"</span>
</pre></div>
</div>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_repeat"
title="ecflow.Node.add_repeat"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_repeat</span></tt></a>, <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Repeat"
title="ecflow.Repeat"><tt ><span
>ecflow.Repeat</span></tt></a>, <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatDate"
title="ecflow.RepeatDate"><tt ><span
>ecflow.RepeatDate</span></tt></a>, <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatEnumerated"
title="ecflow.RepeatEnumerated"><tt class="xref py py-class docutils
literal"><span >ecflow.RepeatEnumerated</span></tt></a>, <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatInteger"
title="ecflow.RepeatInteger"><tt class="xref py py-class docutils
literal"><span >ecflow.RepeatInteger</span></tt></a>, <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatDay"
title="ecflow.RepeatDay"><tt ><span
>ecflow.RepeatDay</span></tt></a>
For text BNF see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-repeat"><tt
><span
>repeat</span></tt></a></p>
</dd>
<dt id="term-running">running</dt>
<dd>Is a <a href="#term-ecflow-server"><em
>ecflow_server</em></a> state. See <a class="reference
internal" href="#term-server-states"><em >server
states</em></a></dd>
<dt id="term-scheduling">scheduling</dt>
<dd><p >The <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
is responsible for <a href="#term-task"><em
>task</em></a> scheduling.</p>
<p >It will check <a
href="#term-dependencies"><em >dependencies</em></a>
in the <a href="#term-suite-definition"><em
>suite definition</em></a> every minute.
If these <a href="#term-dependencies"><em
>dependencies</em></a> are free, the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> will submit the task.
See <a href="#term-job-creation"><em class="xref std
std-term">job creation</em></a>.</p>
</dd>
<dt id="term-server-states">server states</dt>
<dd><p >The following tables reflects the <a class="reference
internal" href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a> capabilities in the different states</p>
<table border="1" >
<colgroup>
<col width="22%" />
<col width="16%" />
<col width="16%" />
<col width="19%" />
<col width="26%" />
</colgroup>
<thead valign="bottom">
<tr ><th >State</th>
<th >User Request</th>
<th >Task Request</th>
<th >Job Scheduling</th>
<th >Auto-Check-pointing</th>
</tr>
</thead>
<tbody valign="top">
<tr ><td><a href="#term-running"><em
>running</em></a></td>
<td>yes</td>
<td>yes</td>
<td>yes</td>
<td>yes</td>
</tr>
<tr ><td><a href="#term-shutdown"><em
>shutdown</em></a></td>
<td>yes</td>
<td>yes</td>
<td>no</td>
<td>yes</td>
</tr>
<tr ><td><a href="#term-halted"><em
>halted</em></a></td>
<td>yes</td>
<td>no</td>
<td>no</td>
<td>no</td>
</tr>
</tbody>
</table>
</dd>
<dt id="term-shutdown">shutdown</dt>
<dd>Is a <a href="#term-ecflow-server"><em
>ecflow_server</em></a> state. See <a class="reference
internal" href="#term-server-states"><em >server
states</em></a></dd>
<dt id="term-status">status</dt>
<dd><p >Each <a href="#term-node"><em
>node</em></a> in <a
href="#term-suite-definition"><em >suite
definition</em></a> has a status.</p>
<p>Status reflects the state of the <a
href="#term-node"><em >node</em></a>.
In <a href="#term-ecflowview"><em class="xref std
std-term">ecflowview</em></a> the background colour of the text reflects the
status.</p>
<p><a href="#term-task"><em class="xref std
std-term">task</em></a> status are: <a
href="#term-unknown"><em >unknown</em></a>, <a
href="#term-queued"><em class="xref std
std-term">queued</em></a>, <a
href="#term-submitted"><em >submitted</em></a>, <a
href="#term-active"><em class="xref std
std-term">active</em></a>, <a
href="#term-complete"><em >complete</em></a>, <a
href="#term-aborted"><em class="xref std
std-term">aborted</em></a> and <a
href="#term-suspended"><em >suspended</em></a></p>
<p ><a href="#term-ecflow-server"><em
>ecflow_server</em></a> status are: <a
href="#term-shutdown"><em class="xref std
std-term">shutdown</em></a>, <a
href="#term-halted"><em >halted</em></a>, <a
href="#term-running"><em class="xref std
std-term">running</em></a>
this is shown on the root node in <a
href="#term-ecflowview"><em >ecflowview</em></a></p>
</dd>
<dt id="term-submitted">submitted</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> <a
href="#term-status"><em >status</em></a>.</p>
<p >When the <a href="#term-task"><em
>task</em></a> <a
href="#term-dependencies"><em >dependencies</em></a>
are resolved/free the <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
places the task into a submitted state.
However if the ECF_JOB_CMD fails, the task is placed into the <a
href="#term-aborted"><em class="xref std
std-term">aborted</em></a> state</p>
</dd>
<dt id="term-suite">suite</dt>
<dd><p >A suite is organisational entity. It is serves as the root
<a href="#term-node"><em class="xref std
std-term">node</em></a> in a <a
href="#term-suite-definition"><em >suite
definition</em></a>.
It should be used to hold a set of jobs that achieve a common function. It can
be used to hold
user <a href="#term-variable"><em class="xref std
std-term">variable</em></a> s that are common to all of its children.</p>
<p>Only a suite node can have a <a
href="#term-clock"><em >clock</em></a>.</p>
<p >It is a collection of <a
href="#term-family"><em >family</em></a>‘s, <a
href="#term-variable"><em class="xref std
std-term">variable</em></a>‘s, <a
href="#term-repeat"><em >repeat</em></a> and a single
<a href="#term-clock"><em class="xref std
std-term">clock</em></a> definition. For a complete list of attributes look at
BNF for <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-suite"><tt
><span
>suite</span></tt></a>.
For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Suite"
title="ecflow.Suite"><tt ><span
>ecflow.Suite</span></tt></a>.</p>
</dd>
<dt id="term-suite-definition">suite definition</dt>
<dd><p >The suite definition is the hierarchical <a
href="#term-node"><em class="xref std
std-term">node</em></a> tree.</p>
<p>It describes how your <a href="#term-task"><em
>task</em></a>‘s run and interact.</p>
<p>It can built up using:</p>
<blockquote>
<div><ul>
<li><p >Ascii text file by following the rules defined in the
ecFlow <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammer"><em>Definition file
Grammar</em></a>.</p>
<p>Hence any language can be used, to generate this format.</p>
</li>
<li><p ><a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#suite-definition-python-api"><em>Suite
Definition API</em></a></p>
</li>
</ul>
</div></blockquote>
<p >Once the definition is built, it can be loaded into the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>, and started.
It can be monitored by <a
href="#term-ecflowview"><em >ecflowview</em></a></p>
</dd>
<dt id="term-suspended">suspended</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> state. A <a
href="#term-node"><em >node</em></a> can be placed
into the suspended state via a <a
href="#term-defstatus"><em >defstatus</em></a> or via
<a href="#term-ecflowview"><em class="xref std
std-term">ecflowview</em></a></p>
<p >A suspended <a href="#term-node"><em
>node</em></a> including any of its children can not
take part in <a href="#term-scheduling"><em
>scheduling</em></a> until
the node is resumed.</p>
</dd>
<dt id="term-task">task</dt>
<dd><p >A task represents a job that needs to be carried out.
It serves as a leaf <a href="#term-node"><em
>node</em></a> in a <a
href="#term-suite-definition"><em >suite
definition</em></a></p>
<p>Only tasks can be submitted.</p>
<p>A job inside a task <a
href="#term-ecf-script"><em >ecf script</em></a>
should generally be re-entrant
so that no harm is done by rerunning it, since a task may be automatically
submitted more than once if it aborts.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Task"
title="ecflow.Task"><tt ><span
>ecflow.Task</span></tt></a>. For text BNF see <a class="reference
internal"
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-task"><tt
><span
>task</span></tt></a></p>
</dd>
<dt id="term-time">time</dt>
<dd><p >This defines a time dependency for a node.</p>
<p>Time is expressed in the format [h]h:mm.
Only numeric values are allowed. There can be multiple time dependencies for a
node,
but overlapping times may cause unexpected results.
To define a series of times, specify the start time, end time and a time
increment.
If the start time begins with ‘+’, times are relative to the
beginning of the suite or,
in repeated families, relative to the beginning of the repeated family.</p>
<p>If the time the job takes to complete is longer than the interval a “slot”
is missed,
e.g.</p>
<div ><pre>time 10:00 20:00 01:00</pre>
</div>
<p>if the 10:00 run takes more than an hour, the 11:00 run will never
occur.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Time"
title="ecflow.Time"><tt ><span
>ecflow.Time</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_time"
title="ecflow.Node.add_time"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_time</span></tt></a>. For BNF see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-time"><tt
><span
>time</span></tt></a></p>
</dd>
<dt id="term-today">today</dt>
<dd><p >Like <a href="#term-time"><em
>time</em></a>, but If the suites begin time is
<strong>past</strong> the time given for the “today” command the <a
href="#term-node"><em class="xref std
std-term">node</em></a> is free
to run (as far as the time dependency is concerned).</p>
<p>For example</p>
<div ><pre>task x
today 10:00</pre>
</div>
<p>If we begin or re-queue the <a
href="#term-suite"><em >suite</em></a> at 9.00 am,
then the <a href="#term-task"><em class="xref std
std-term">task</em></a> in held until 10.00 am.
However if we begin or re-queue the suite at 11.00am, the <a class="reference
internal" href="#term-task"><em >task</em></a> is run
immediately.</p>
<p>No lets look at time</p>
<div ><pre>task x
time 10:00</pre>
</div>
<p>If we begin or re-queue the <a
href="#term-suite"><em >suite</em></a> at 9.00am, then
the <a href="#term-task"><em class="xref std
std-term">task</em></a> in held until 10.00 am.
If we begin or re-queue the <a
href="#term-suite"><em >suite</em></a> at 11.00am, the
<a href="#term-task"><em class="xref std
std-term">task</em></a> is still held.</p>
<p>If the time the job takes to complete is longer than the interval a “slot”
is missed,
e.g.</p>
<div ><pre>today 10:00 20:00 01:00</pre>
</div>
<p>if the 10:00 run takes more than an hour, the 11:00 run will never
occur.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Today"
title="ecflow.Today"><tt ><span
>ecflow.Today</span></tt></a>. For text BNF see <a class="reference
internal"
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-today"><tt
><span
>today</span></tt></a></p>
</dd>
<dt id="term-trigger">trigger</dt>
<dd><p >Triggers defines a dependency for a <a class="reference
internal" href="#term-task"><em >task</em></a> or <a
href="#term-family"><em class="xref std
std-term">family</em></a>.</p>
<p>There can be only one trigger dependency per <a
href="#term-node"><em >node</em></a>,
but that can be a complex boolean expression of the <a class="reference
internal" href="#term-status"><em >status</em></a> of
several nodes.
Triggers should be avoided on suites.
A node with a trigger can only be activated when its trigger has expired.
A trigger holds the node as long as the trigger’s expression evaluation
returns false.</p>
<p>Trigger evaluation occurs when ever the <a
href="#term-child-command"><em >child command</em></a>
communicates with the server. i.e whenever
there is a state change in the suite definition.</p>
<p>The keywords in trigger expressions are: <a
href="#term-unknown"><em >unknown</em></a>, <a
href="#term-suspended"><em class="xref std
std-term">suspended</em></a>, <a
href="#term-complete"><em >complete</em></a>, <a
href="#term-queued"><em class="xref std
std-term">queued</em></a>, <a
href="#term-submitted"><em >submitted</em></a>, <a
href="#term-active"><em class="xref std
std-term">active</em></a>, <a
href="#term-aborted"><em >aborted</em></a>
and <strong>clear</strong> and <strong>set</strong> for <a class="reference
internal" href="#term-event"><em >event</em></a>
status.</p>
<p>Triggers can also reference Node attributes like <a class="reference
internal" href="#term-event"><em >event</em></a>, <a
href="#term-meter"><em class="xref std
std-term">meter</em></a>, <a
href="#term-variable"><em >variable</em></a>, <a
href="#term-repeat"><em class="xref std
std-term">repeat</em></a> and generated variables.
Trigger evaluation for node attributes uses integer arithmetic:</p>
<ul >
<li><a href="#term-event"><em class="xref std
std-term">event</em></a> has the integer value of 0(clear) and set(1)</li>
<li><a href="#term-meter"><em class="xref std
std-term">meter</em></a> values are integers hence they are used as is</li>
<li><a href="#term-variable"><em class="xref std
std-term">variable</em></a> value is converted to an integer, otherwise 0 is
used. See example below</li>
<li><a href="#term-repeat"><em class="xref std
std-term">repeat</em></a> <em>string</em> : We use the index values as
integers. See example below</li>
<li><a href="#term-repeat"><em class="xref std
std-term">repeat</em></a> <em>enumerated</em> : We use the index values as
integers. See example below</li>
<li><a href="#term-repeat"><em class="xref std
std-term">repeat</em></a> <em>integer</em> : Use the implicit integer
values</li>
<li><a href="#term-repeat"><em class="xref std
std-term">repeat</em></a> <em>date</em> : Use the date values as
integers. Use of plus/minus on repeat date variable uses date arithmetic</li>
</ul>
<p>Here are some examples</p>
<div ><pre>suite trigger_suite
task a
event EVENT
meter METER 1 100 50
edit VAR_INT 12
edit VAR_STRING "captain scarlett" # This is not convertible to
an integer, if referenced will use '0'
family f1
edit SLEEP 2
repeat string NAME a b c d e f # This has values: a(0),
b(1), c(3), d(4), e(5), f(6) i.e index
family f2
repeat integer VALUE 5 10 # This has values:
5,6,7,8,9,10
family f3
repeat enumerated red green blue # red(0), green(1), blue(2)
task t1
repeat date DATE 19991230 20000102 # This has values:
19991230,19991231,20000101,20000102
endfamily
endfamily
endfamily
family f2
task event_meter
trigger /suite/a:EVENT == set and /suite/a:METER >= 30
task variable
trigger /suite/a:VAR_INT >= 12 and /suite/a:VAR_STRING == 0
task repeat_string
trigger /suite/f1:NAME >= 4
task repeat_integer
trigger /suite/f1/f2:VALUE >= 7
task repeat_date
trigger /suite/f1/f2/f3/t1:DATE >= 19991231
task repeat_date2
# Using plus/minus on a repeat DATE will use date arithmetic
# Since the starting value of DATE is 19991230, this task will run
# straight away
trigger /suite/f1/f2/f3/t1:DATE - 1 == 19991229
endfamily
endsuite</pre>
</div>
<p>What happens when we have multiple node attributes of the same name,
referenced in trigger expressions ?</p>
<div ><pre>task foo
event blah
meter blah 0 200 50
edit blah 10
task bar
trigger foo:blah >= 0</pre>
</div>
<p>In this case ecFlow will use the following precedence:</p>
<ul >
<li><a href="#term-event"><em class="xref std
std-term">event</em></a></li>
<li><a href="#term-meter"><em class="xref std
std-term">meter</em></a></li>
<li><a href="#term-variable"><em class="xref std
std-term">variable</em></a></li>
<li><a href="#term-repeat"><em class="xref std
std-term">repeat</em></a></li>
<li>generated variables</li>
</ul>
<p>Hence in the example above expression ‘foo:blah >= 0’ will
reference the event.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Expression"
title="ecflow.Expression"><tt ><span
>ecflow.Expression</span></tt></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_trigger"
title="ecflow.Node.add_trigger"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_trigger</span></tt></a></p>
</dd>
<dt id="term-unknown">unknown</dt>
<dd><p >Is a <a href="#term-node"><em
>node</em></a> <a
href="#term-status"><em >status</em></a>.</p>
<p >This is the default <a
href="#term-node"><em >node</em></a> <a
href="#term-status"><em class="xref std
std-term">status</em></a> when a <a
href="#term-suite-definition"><em >suite
definition</em></a> is loaded into the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></p>
</dd>
<dt id="term-user-commands">user commands</dt>
<dd>User commands are any client to server requests that are
<strong>not</strong> <a
href="#term-child-command"><em >child command</em></a>
s.</dd>
<dt id="term-variable">variable</dt>
<dd><p >ecFlow makes heavy use of different kinds of
variables.There are several kinds of variables:</p>
<blockquote>
<div><p>Environment variables: which are set in the UNIX shell before the <a
href="#term-ecflow"><em class="xref std
std-term">ecFlow</em></a> starts.
These control <a href="#term-ecflow-server"><em
>ecflow_server</em></a>, and <a class="reference
internal" href="#term-ecflow-client"><em class="xref std
std-term">ecflow_client</em></a> .</p>
<p>suite definition variables: Also referred to as user variables. These
control <a href="#term-ecflow-server"><em
>ecflow_server</em></a>, and <a class="reference
internal" href="#term-ecflow-client"><em class="xref std
std-term">ecflow_client</em></a> and
are available for use in <a
href="#term-job-file"><em >job file</em></a>.</p>
<p>Generated variables: These are generated within the <a class="reference
internal" href="#term-suite-definition"><em >suite
definition</em></a> node tree during <a
href="#term-job-creation"><em >job creation</em></a>
and are available for use in the <a
href="#term-job-file"><em >job file</em></a>.</p>
</div></blockquote>
<p>Variables can be referenced in <a
href="#term-trigger"><em >trigger</em></a> and <a
href="#term-complete-expression"><em class="xref std
std-term">complete expression</em></a> s . The value part of the variable
should
be convertible to an integer otherwise a default value of 0 is used.</p>
<p >For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_variable"
title="ecflow.Node.add_variable"><tt class="xref py py-class docutils
literal"><span >ecflow.Node.add_variable</span></tt></a>. For BNF
see <a
href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-variable"><tt
><span
>variable</span></tt></a></p>
</dd>
<dt id="term-variable-inheritance">variable inheritance</dt>
<dd><p >When a <a
href="#term-variable"><em >variable</em></a> is needed
at <a href="#term-job-creation"><em class="xref std
std-term">job creation</em></a> time, it is first sought in the <a
href="#term-task"><em class="xref std
std-term">task</em></a> itself.</p>
<p>If it is not found in the <a
href="#term-task"><em >task</em></a>, it is sought
from the task’s parent and so on, up through
the <a href="#term-node"><em class="xref std
std-term">node</em></a> levels until found.</p>
<p>For any <a href="#term-node"><em class="xref std
std-term">node</em></a>, there are two places to look for variables.</p>
<p >Suite definition variables are looked for first, and then any
generated variables.</p>
</dd>
<dt id="term-variable-substitution">variable substitution</dt>
<dd><p >Takes place during <a
href="#term-pre-processing"><em class="xref std
std-term">pre-processing</em></a> <em>or</em> command invocation.(i.e
ECF_JOB_CMD,ECF_KILL_CMD,etc)</p>
<p>It involves searching each line of <a
href="#term-ecf-script"><em >ecf script</em></a> file
or command, for <a href="#term-ecf-micro"><em
>ECF_MICRO</em></a> character. typically
‘%’</p>
<p>The text between two % character, defines a variable. i.e %VAR%</p>
<p>This variable is searched for in the <a
href="#term-suite-definition"><em >suite
definition</em></a>.</p>
<p>First the suite definition variables( sometimes referred to as user
variables) are searched
and then Repeat variable name, and finally the generated variables.If no
variable
is found then the same search pattern is repeated up the node tree.</p>
<p>The value of the <a href="#term-variable"><em
>variable</em></a> is replaced between the %
characters.</p>
<p>If the micro character are not paired and an error message is written to the
log file,
and the task is placed into the <a
href="#term-aborted"><em >aborted</em></a> state.</p>
<p>If the variable is not found in the <a
href="#term-suite-definition"><em >suite
definition</em></a> during pre-processing then <a
href="#term-job-creation"><em >job creation</em></a>
fails,
and an error message is written to the log file, and the task is placed into
the <a href="#term-aborted"><em class="xref std
std-term">aborted</em></a> state.
To avoid this, variables in the <a
href="#term-ecf-script"><em >ecf script</em></a> can
be defined as:</p>
<div ><pre>%VAR:replacement%</pre>
</div>
<p >This is similar to %VAR% but if VAR is not found in the <a
href="#term-suite-definition"><em class="xref std
std-term">suite definition</em></a> then ‘replacement’ is used.</p>
</dd>
<dt id="term-virtual-clock">virtual clock</dt>
<dd><p >Like <a
href="#term-real-clock"><em >real clock</em></a> until
the <a href="#term-ecflow-server"><em class="xref
std std-term">ecflow_server</em></a> is suspended (i.e <a class="reference
internal" href="#term-shutdown"><em >shutdown</em></a>
or <a href="#term-halted"><em class="xref std
std-term">halted</em></a>),
the suites <a href="#term-clock"><em class="xref std
std-term">clock</em></a> is also suspended.</p>
<p>Hence will honour relative times in <a
href="#term-cron"><em >cron</em></a>, <a
href="#term-today"><em class="xref std
std-term">today</em></a> and <a
href="#term-time"><em >time</em></a> dependencies.
It is possible to have a combination of hybrid/real and virtual.</p>
<p >More useful when we want complete adherence to time related
dependencies at the expense
being out of sync with system time.</p>
</dd>
<dt id="term-zombie">zombie</dt>
<dd><p >Zombies are running jobs that fail authentication when
communicating with the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></p>
<p><a href="#term-child-command"><em class="xref std
std-term">child command</em></a> s like (init, event,meter, label,
abort,complete) are placed in the <a
href="#term-ecf-script"><em >ecf script</em></a>
file and are used to communicate with the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>.</p>
<p>The <a href="#term-ecflow-server"><em class="xref
std std-term">ecflow_server</em></a> authenticates each connection attempt made
by the <a href="#term-child-command"><em class="xref
std std-term">child command</em></a>.
Authentication can fail for a number of reasons:</p>
<blockquote>
<div><ul >
<li>password(ECF_PASS) supplied with the <a
href="#term-child-command"><em >child
command</em></a>, does not match the one in the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></li>
<li>path name(ECF_NAME) supplied with the <a
href="#term-child-command"><em >child
command</em></a>, does not locate a <a
href="#term-task"><em >task</em></a> in the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></li>
<li>process id(ECF_RID) supplied with <a
href="#term-child-command"><em >child
command</em></a>, does not correspond with the one stored in the <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></li>
<li><a href="#term-task"><em class="xref std
std-term">task</em></a> is already <a
href="#term-active"><em >active</em></a>, but receives
another init <a href="#term-child-command"><em
>child command</em></a></li>
<li><a href="#term-task"><em class="xref std
std-term">task</em></a> is already <a
href="#term-complete"><em >complete</em></a>, but
receives another <a href="#term-child-command"><em
>child command</em></a></li>
</ul>
</div></blockquote>
<p>When authentication fails the job is considered to be a zombie.
The <a href="#term-ecflow-server"><em class="xref
std std-term">ecflow_server</em></a> will keep a note of the zombie for a
period of time, before it is automatically removed.
However the removed zombie, may well re-appear. ( this is because each <a
href="#term-child-command"><em class="xref std
std-term">child command</em></a> will continue
attempting to contact the <a
href="#term-ecflow-server"><em >ecflow_server</em></a>
for 24 hours. This is configurable
see ECF_TIMEOUT on <a href="#term-ecflow-client"><em
>ecflow_client</em></a>)</p>
<p>For python see <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr"
title="ecflow.ZombieAttr"><tt ><span
>ecflow.ZombieAttr</span></tt></a>, <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieUserActionType"
title="ecflow.ZombieUserActionType"><tt class="xref py py-class docutils
literal"><span >ecflow.ZombieUserActionType</span></tt></a></p>
<p >There are several types of zombies see <a class="reference
internal" href="#term-zombie-type"><em >zombie
type</em></a> and <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieType"
title="ecflow.ZombieType"><tt ><span
>ecflow.ZombieType</span></tt></a></p>
</dd>
<dt id="term-zombie-attribute">zombie attribute</dt>
<dd><p >The zombie attribute defines how a <a class="reference
internal" href="#term-zombie"><em >zombie</em></a>
should be handled in an automated fashion.
Very careful consideration should be taken before this attribute is added as it
may hide a genuine problem.
It can be added to any <a href="#term-node"><em
>node</em></a>. But is best defined at the <a
href="#term-suite"><em class="xref std
std-term">suite</em></a> or <a
href="#term-family"><em >family</em></a> level.
If there is no zombie attribute the default behaviour is to block the <a
href="#term-child-command"><em class="xref std
std-term">child command</em></a>.</p>
<p >To add a zombie attribute in python, please see: <a
href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr"
title="ecflow.ZombieAttr"><tt ><span
>ecflow.ZombieAttr</span></tt></a></p>
</dd>
<dt id="term-zombie-type">zombie type</dt>
<dd><p >See <a href="#term-zombie"><em
>zombie</em></a> and class <a class="reference
internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr"
title="ecflow.ZombieAttr"><tt ><span
>ecflow.ZombieAttr</span></tt></a> for further information.
There are several types of zombies:</p>
<blockquote>
<div><ul>
<li><dl >
<dt>path</dt>
<dd><ul >
<li>The task path can not be found in the server, because node tree was
deleted,
replaced,reload, server crashed or backup server does not have node tree.</li>
<li>Jobs could have been created, via server <a
href="#term-scheduling"><em >scheduling</em></a> or by
<a href="#term-user-commands"><em class="xref std
std-term">user commands</em></a></li>
</ul>
</dd>
</dl>
</li>
<li><dl >
<dt>user</dt>
<dd><p >Job is created by <a
href="#term-user-commands"><em >user commands</em></a>
like, rerun, re-queue. User zombies are
differentiated from server(scheduled) since they are automatically created when
the force option is used and we have tasks in an <a
href="#term-active"><em >active</em></a> or <a
href="#term-submitted"><em class="xref std
std-term">submitted</em></a> states.</p>
</dd>
</dl>
</li>
<li><dl >
<dt>ecf</dt>
<dd><p >Jobs are created as part of the normal <a class="reference
internal" href="#term-scheduling"><em class="xref std
std-term">scheduling</em></a></p>
<ul >
<li>Server crashed ( or terminated and restarted) and the recovered <a
href="#term-check-point"><em class="xref std
std-term">check point</em></a> file is out of date.</li>
<li>A <a href="#term-task"><em class="xref std
std-term">task</em></a> is repeatedly re-run, earlier copies will not be
remembered.</li>
<li>Job sent by another <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a>, but which can not talk to the original <a
href="#term-ecflow-server"><em class="xref std
std-term">ecflow_server</em></a></li>
<li>Network glitches/network down</li>
</ul>
</dd>
</dl>
</li>
</ul>
</div></blockquote>
<p >The type of the zombie is not fixed and may change.</p>
</dd>
</dl>
</div>
|
...