Horizontal Navigation Bar |
---|
Button Group |
---|
Button Hyperlink |
---|
title | Previous |
---|
type | standard |
---|
url | https://confluence.ecmwf.int/display/ECFLOW/Variable+inheritance |
---|
|
Button Hyperlink |
---|
title | Up |
---|
type | standard |
---|
url | https://confluence.ecmwf.int/display/ECFLOW/Going+Further |
---|
|
Button Hyperlink |
---|
title | Next |
---|
type | standard |
---|
url | https://confluence.ecmwf.int/display/ECFLOW/Embedded+Triggers |
---|
|
|
|
In the previous exercise, we saw that the two tasks were running simultaneously.We would like now to make sure that t2 only runs once t1 is complete.
For this, we have to define a trigger Triggers are used to declare dependencies between two tasks. For instance, the second task might need data created by the first task.
When ecFlow tries to start a task, it evaluates the trigger expression. If the condition is correct, the task is started, otherwise, the task stays queued. Triggers can be between tasks, or between families, or a mixture.
Remember the two rules:
- A family is complete when all its tasks are complete
- A task will be started if its triggers and the triggers of all its parent families evaluate to true
A node can only have one trigger expression, but very complex expressions can be built (and keep in mind that the triggers of the parent nodes are also implicit triggers). Sometimes triggers are also used to prevent too many jobs from running at the same time. In this case, the use of a limit may be a better solution (we will cover limits later on, in the Limits section). Nodes can be addressed in trigger expressions using full names: /test/f1/t1 refers to the task t1, and /test/f1 refers to the family f1. In some contexts, ecFlow will accept relative names, such as ../t1.
trigger /test/f1/t1 == complete
Triggers can be very complex, and ecFlow supports all kinds of conditions
(not, and, or, ...), in addition, they can also reference Node attributes like
...
Definition of the suite test.
suite test
edit ECF_INCLUDE "$HOME/course" # replace '$HOME' with the path to your home directory
edit ECF_HOME "$HOME/course"
family f1
edit SLEEP 20
task t1
task t2
trigger t1 eq complete
endfamily
endsuite |
Python
The trigger expression can be checked, this is especially important when dealing with very large suites and complex triggers.
Code Block |
---|
language | py |
---|
title | $HOME/course/test.py |
---|
|
import os
from ecflow import Defs,Suite,Family,Task,Edit,Trigger
def create_family_f1():
return Family("f1",
Edit(SLEEP=20),
Task("t1"),
Task("t2",Trigger("t1 == complete")))
print("Creating suite definition")
home = os.path.join(os.getenv("HOME"), "course")
defs = Defs(
Suite("test",
Edit(ECF_INCLUDE=home,ECF_HOME=home),
create_family_f1()))
print(defs)
print("check trigger expressions")
check = defs.check()
assert len(check) == 0, check
print("Checking job creation: .ecf -> .job0")
print(defs.check_job_creation())
print("Saving definition to file 'test.def'")
defs.save_as_defs("test.def") |
What to do
- Edit the suite definition file to add the trigger.
- Replace the suite
python: python3 test.py ; python3 client.py
text: ecflow_client --suspend=/test ; ecflow_client --replace=/test test.def - Observe the tasks in ecflow_ui .
- See the triggers by selecting t1 or t2.
- See the trigger relation by clicking on the trigger tab.
- Search any reference to t1 by using the search menu.
Introduce an error in the trigger expression and ensure that this error is trapped. i.e. change the trigger to.
Code Block |
---|
language | py |
---|
title | Check trigger expressions |
---|
|
Trigger("t == complete") # there is no node with name t, this should be reported as an error |
Horizontal Navigation Bar |
---|
Button Group |
---|
Button Hyperlink |
---|
title | Previous |
---|
type | standard |
---|
url | https://confluence.ecmwf.int/display/ECFLOW/Variable+inheritance |
---|
|
Button Hyperlink |
---|
title | Up |
---|
type | standard |
---|
url | https://confluence.ecmwf.int/display/ECFLOW/Going+Further |
---|
|
Button Hyperlink |
---|
title | Next |
---|
type | standard |
---|
url | https://confluence.ecmwf.int/display/ECFLOW/Embedded+Triggers |
---|
|
| endsuite</pre>
</div>
<p>In python this would be:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c">##!/usr/bin/env python2.5</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">ecflow</span>
<span class="k">def</span> <span class="nf">create_family_f1</span><span class="p">():</span>
<span class="n">f1</span> <span class="o">=</span> <span class="n">ecflow</span><span class="o">.</span><span class="n">Family</span><span class="p">(</span><span class="s">"f1"</span><span class="p">)</span>
<span class="n">f1</span><span class="o">.</span><span class="n">add_variable</span><span class="p">(</span><span class="s">"SLEEP"</span><span class="p">,</span><span class="mi">20</span><span class="p">)</span>
<span class="n">f1</span><span class="o">.</span><span class="n">add_task</span><span class="p">(</span><span class="s">"t1"</span><span class="p">)</span>
<span class="n">t2</span> <span class="o">=</span> <span class="n">f1</span><span class="o">.</span><span class="n">add_task</span><span class="p">(</span><span class="s">"t2"</span><span class="p">)</span>
<span class="n">t2</span><span class="o">.</span><span class="n">add_trigger</span><span class="p">(</span><span class="s">"t1 eq complete"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">f1</span>
<span class="n">defs</span> <span class="o">=</span> <span class="n">ecflow</span><span class="o">.</span><span class="n">Defs</span><span class="p">()</span>
<span class="n">suite</span> <span class="o">=</span> <span class="n">defs</span><span class="o">.</span><span class="n">add_suite</span><span class="p">(</span><span class="s">"test"</span><span class="p">)</span>
<span class="n">suite</span><span class="o">.</span><span class="n">add_variable</span><span class="p">(</span><span class="s">"ECF_INCLUDE"</span><span class="p">,</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"HOME"</span><span class="p">)</span> <span class="o">+</span> <span class="s">"/course"</span><span class="p">)</span>
<span class="n">suite</span><span class="o">.</span><span class="n">add_variable</span><span class="p">(</span><span class="s">"ECF_HOME "</span><span class="p">,</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"HOME"</span><span class="p">)</span> <span class="o">+</span> <span class="s">"/course"</span><span class="p">)</span>
<span class="n">suite</span><span class="o">.</span><span class="n">add_family</span><span class="p">(</span> <span class="n">create_family_f1</span><span class="p">()</span> <span class="p">)</span>
</pre></div>
</div>
<div class="line-block">
<div class="line">Triggers are used to declare <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-dependencies"><em class="xref std std-term">dependencies</em></a> between two tasks.</div>
<div class="line">For instance, the second task might need data created by the first task.</div>
</div>
<div class="line-block">
<div class="line">When ecFlow tries to start a task, it evaluates the <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-trigger"><em class="xref std std-term">trigger</em></a> expression.</div>
<div class="line">If the condition is correct, the task is started, otherwise the task</div>
<div class="line">stays <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-queued"><em class="xref std std-term">queued</em></a>.</div>
</div>
<div class="line-block">
<div class="line">Triggers can be between tasks, or between families, or a mixture.</div>
<div class="line">Remember the two rules:</div>
</div>
<ul class="simple">
<li>A family is <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-complete"><em class="xref std std-term">complete</em></a> when all its tasks are <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-complete"><em class="xref std std-term">complete</em></a>.</li>
<li>A task will be started if its triggers and the triggers of all is
parent families evaluate to true.</li>
</ul>
<div class="line-block">
<div class="line">A <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-node"><em class="xref std std-term">node</em></a> can only have one trigger expression, but very complex</div>
<div class="line">expressions can be built (and keep in mind that the triggers of the</div>
<div class="line">parent nodes are also implicit triggers).</div>
</div>
<div class="line-block">
<div class="line">Sometimes triggers are also used to prevent too many jobs from running</div>
<div class="line">at the same time. In this case the use of a <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-limit"><em class="xref std std-term">limit</em></a> may be a better</div>
<div class="line">solution (we will cover limits later on in the <a class="reference internal" href="/wiki/display/ECFLOW/Limits#limits"><em>Limits</em></a> section).</div>
</div>
<div class="line-block">
<div class="line">Triggers can be very complex, and ecFlow supports all kinds of conditions</div>
<div class="line">(not, and, or, ...)</div>
</div>
<p>What to do:</p>
<ol class="arabic simple">
<li>Edit the <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-suite-definition"><em class="xref std std-term">suite definition</em></a> file to add the <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-trigger"><em class="xref std std-term">trigger</em></a>.</li>
<li>Play the definition again.</li>
<li>Observe the tasks in <a class="reference internal" href="/wiki/display/ECFLOW/Glossary#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>.</li>
<li>See the triggers by selecting <strong>t1</strong> or <strong>t2</strong> and clicking on the <img alt="triggers" src="../../../_images/triggers.jpg" /> icon.</li>
<li>See the trigger relation by clicking on the arrow.</li>
<li>See the triggers in the tree, using the Show menu.</li>
<li>Search any reference to <strong>t1</strong> by using the <img alt="search" src="../../../_images/search.jpg" /> icon</li>
</ol>
</div>
|