| HTML |
|---|
<div class="section" id="example"> <span id="index-0"></span><span id="id1"></span> <p>A suite template is provided to demonstrate basic SMS features which can be used as a starting point for suite design.</p> <ul> <li><p class="first">Download the tar-file <a class="reference download internal" href="../../_downloads/skull.tgz"><tt class="xref download docutils literal"><span class="pre">skull</span> <span class="pre">suite</span></tt></a></p> </li> <li><p class="first">untar it:</p> <div class="highlight-python"><pre>> tar -xzvf skull.tgz</pre> </div> </li> <li><p class="first">eventually, start a SMS server:</p> <div class="highlight-python"><pre>> sms_start</pre> </div> </li> <li><p class="first">convention at ECMWF is to attribute a program number according to the formula:</p> <div class="highlight-python"><pre>> prog_num=$((900000 + $(id -u))); echo $prog_num</pre> </div> </li> <li><p class="first">open xcdp, File->Login to: “localhost <prog_num>”</p> </li> <li><p class="first">play the skull suite into sms:</p> <div class="highlight-python"><pre>> export SMS_PROG=$((900000+$(id -u))) > LOGGER="setenv -i $SMS_PROG; login localhost $USER 1;" > /usr/local/apps/sms/bin/cdp -c "$LOGGER play -r /skull skull.def"</pre> </div> </li> <li><p class="first">skull suite families are</p> <ul class="simple"> <li>make: to open a xterm window from jobs run on local workstation, ecgate, HPC, linux cluter</li> <li>consumer: to demonstrate the producer-consumer pattern</li> <li>dinner: a resource sharing example through the dinning philosophers example</li> <li>shop: to practice dynamic tasks generation</li> <li>perl, Python: as task template examples that do not use ksh</li> </ul> <img alt="../../_images/skull.png" src="../../_images/wiki/download/attachments/7373012/skull.png" /> </li> <li><p class="first">conversion to a text based definition file written by ksh (skull.sh) is rather straightfoward:</p> <div class="highlight-python"><pre>> make sh</pre> </div> <p>libgen.sh is generated to provide cdp:play commands in ksh environment:</p> <div class="highlight-python"><pre>> bin/genlibdef.sh sh libgen.sh</pre> </div> <div class="highlight-python"><pre>set -eu if [[ ${DEBUG_DEF2ECF:=0} != 0 ]] ; then set -eux ; fi alias EOF='echo EOF >&3 ' alias abort='echo abort >&3 ' alias action='echo action >&3 ' alias autocancel='echo autocancel >&3 ' alias automigrate='echo automigrate >&3 ' alias autorestore='echo autorestore >&3 ' alias clock='echo clock >&3 ' alias complete='echo complete >&3 ' alias cron='echo cron >&3 ' alias date='echo date >&3 ' alias day='echo day >&3 ' alias defstatus='echo defstatus >&3 ' alias edit='echo edit >&3 ' alias endfamily='echo endfamily >&3 ' alias endsuite='echo endsuite >&3 ' alias endtask='echo endtask >&3 ' alias event='echo event >&3 ' alias extern='echo extern >&3 ' alias family='echo family >&3 ' alias inlimit='echo inlimit >&3 ' alias label='echo label >&3 ' alias late='echo late >&3 ' alias limit='echo limit >&3 ' alias meter='echo meter >&3 ' alias owner='echo owner >&3 ' alias repeat='echo repeat >&3 ' alias suite='echo suite >&3 ' alias task='echo task >&3 ' alias text='echo text >&3 ' alias time='echo time >&3 ' alias today='echo today >&3 ' alias trigger='echo trigger >&3 ' </pre> </div> <p>At this stage, libgen.sh can be manually updated, to replace EOF with exit 0, inhibit the action, automigrate, autorestore, owner commands (which are unavailable in ecFlow). Change the limit, date, time naming, as it interferes with ksh intrinsics.</p> <p>It is possible to transcode most CDP commands into ksh, defining aliases that will print the cdp:play command and redirect their output in a dedicated file descriptor:</p> <div class="highlight-python"><pre>> bin/def2ecf.sh sh gen</pre> </div> <p>sh/skull.sh contains the ksh command to create the file descriptor 3:</p> <div class="highlight-python"><pre>> exec 3> expanded.tmp</pre> </div> <p>SMS <strong>.def</strong> definition files may be updated to facilitate such transcoding (#CONV tags into *.def files):</p> <blockquote> <div><ul> <li><p class="first">commands can be split into multiline commands to simplify mathematical expressions incompatible commands (owner, autorestore, automigrate, action) can be removed or set as a comment</p> </li> <li><p class="first">multiline triggers changed into one line commands</p> </li> <li><p class="first">the heavy use of aliases can lead to confused transcoding</p> </li> <li><p class="first">CDP is known to be “tolerant”: for example, an unboumded endin command is transparent to CDP, but it would mislead the transcoder.</p> </li> <li><p class="first">for CDP, commands accept minimum significant string for options; they should be replace with their unambiguous version:</p> <div class="highlight-python"><pre>> # CDP: repeat int iter 1 $life_expectency > repeat integer iter 1 $life_expectency</pre> </div> </li> <li><p class="first">through its simplicity, the CDP language was a way to keep away from complex code. Once left behind, it is each suite designers responsability to stick to simple manageable, portable and maintainable code.</p> </li> <li><p class="first">beware homonyms: such as date, time, limit</p> </li> </ul> </div></blockquote> <p>This demonstrates the feasibility, for a simple suite. Past experience shows it is not as easy when considering a more complex suite design.</p> <p>The suite definition is then expanded, as a file sh/expanded.tmp:</p> <div class="highlight-python"><pre>> bin/def2ecf.sh sh expand</pre> </div> <p>or:</p> <div class="highlight-python"><pre>> cd sh; ksh skull.sh</pre> </div> </li> <li><p class="first">the suite can be played locally and expanded for comparison:</p> <div class="highlight-python"><pre>> /usr/local/apps/sms/bin/cdp -c "play -l skull.def; show > expand.cdp" > kompare expand.cdp sh/expand.tmp > kompare skull.def sh/skull.sh</pre> </div> </li> <li><p class="first">it can then be loaded into SMS:</p> <div class="highlight-python"><pre>> /usr/local/apps/sms/bin/cdp -c "LOGGER play -r /skull sh/expanded.tmp"</pre> </div> </li> <li><p class="first">apply the command begin and resume with XCdp or CDP</p> </li> <li><p class="first">We can then start ecFlow server, if not already done:</p> <div class="highlight-python"><pre>> use ecflow > export ECF_PORT=$((20000+$(id -u))) > mkdir ecf && cd ecf > ecflow_server</pre> </div> </li> <li><p class="first">expanded suite can be loaded into ecFlow:</p> <div class="highlight-python"><pre>> cd .. > ecflow_client --load sh/expanded.tmp</pre> </div> </li> <li><p class="first">once loaded, it can be replaced with:</p> <div class="highlight-python"><pre>> ecflow_client --replace /skull sh/expanded.tmp</pre> </div> </li> <li><p class="first">commands begin and resume can be applied on the console or with ecflowview:</p> <div class="highlight-python"><pre>> ecflow_client --begin /skull > ecflow_client --resume /skull > ecflow_client --resume /skull/skull /skull/perl /skull/python</pre> </div> </li> <li><p class="first">in this example, it is enough to <strong>link task wrappers</strong> from the original name <em>.sms</em> to the expected name <em>.ecf</em>.</p> <p>Alternatively, ECF_EXTN variable may be defined, as <strong>.sms</strong> on the top node.</p> </li> <li><p class="first"><strong>smsfiles/passby.sms</strong> task wrapper has been updated to be compatible in both modes.</p> </li> <li><p class="first"><strong>smsfiles/perl.sms</strong> and <strong>smsfiles/python.sms</strong> call an intermediate function to access the meter child command: call_meter</p> </li> <li><p class="first">in the directory include: trap.h, perl_header.h, python_header.h, endt.h, were modified to enable ecFlow mode.</p> </li> <li><p class="first"><strong>include/inc_ecf.h</strong> was added and included from trap.h to add new expected variables, when ECF_PORT variable is not 0.</p> </li> <li><p class="first">it may be an interesting exercise to transform into Python code the file skull.def and the related families defined in their dedicated definition files. The script ./bin/def2ecf.pl can be used to produce a transcoded file. Careful attention is required to fix the transcoder mistakes and obtain a script that can be parsed with Python. Then a script that can be loaded as a proper suite into SMS, or ecFlow.</p> <ul class="simple"> <li>pydef_example/inc_hostc.py is a Python library used to maintain a definition file syntax close to CDP’s.</li> <li>the transcoded file has to be checked for all dollar variables</li> <li>all CDP functions parameters are transcoded incorrectly: $1, $2 ... They can be transformed into named arguments, with a default value when needed.</li> <li>the definition tree is created adding the lines</li> </ul> </li> </ul> <dl class="docutils"> <dt>::</dt> <dd><blockquote class="first"> <div>s = suite(SELECTION) DEFS = Defs() DEFS.add_suite(s)</div></blockquote> <ul class="last simple"> <li>choice must be made whether to use “import library” or “from library import *”.</li> <li>to maintain code readability, new functions can be created (create_suite)</li> </ul> </dd> </dl> </div> |
Page History
Overview
Community Forums
Content Tools