A meter can be used to monitor the disk space available in the working directory of the suite's tasks. This will enable the possibility to postpone the execution of space-demanding tasks by defining a trigger condition on that meter. In this simple pattern, a check_disk family contains the meter (named use) while an update_disk task runs periodically, by cron, to modify the value of the meter.
| Code Block | ||||
|---|---|---|---|---|
| ||||
from ecf import Family,Meter,Task,Cron, ...
...
Family("check_disk").add(
Meter("use",-1,100),
Task("update_disk").add( |
An acquisition task may be needed to initiate the processing. The following example shows such a pattern where a wait task is created to check the presence of input data in the acquisition time window 10:00 to 12:00. A task is dedicated to raise a red alarm, by 11:00, if data are not yet arrived, to have operators and analysts aware in advance and careful about this. wait task runs every 5 minutes and set the data event when input data are found ; it shall sleep 60 seconds, after setting the event, before completing, to ensure the server receives the event and starts the data task, before wait task completion and immediate requeue, thanks to cron. data task (validation and preprocessing) will then set the ready event and acq family will become complete, provided underneath tasks are complete or queued, so that rt/wait and late_alert do not run any further.
...
|
...
Cron("00:00 |
...
23: |
...
59 00:05 |
...
In some situation, we may want the acquisition family to detect if data are already available (catchup mode), and start the processing immediately, before reaching the real-time mode, when the wait task is submitted regularly to check data arrival during the time interval.
"))
... |
| Code Block | ||||
|---|---|---|---|---|
| ||||
#!/usr/bin/ksh
%manual
DESCRIPTION
Check available disk space in working directory's fileset and update meter
%end
%include <init.h>
value=$(df %WORKDIR% | grep %WORKDIR% | awk '{print $5}' | sed s/%%//g)
ecflow_client --alter=change meter "use" "$value" /%SUITE%/%FAMILY%
%include <endt.h>
|
This pattern has been used, for instance, in a suite that reads re-analysis data from mars and re-archives it with a different experiment version number. This sort of processing is inherent parallel but, quite often, it requires the creation of files to temporarily store the data. So it is a common case that the amount of disk space available for those files is the main constraint that limits the parallelism of the process.
| Gliffy Diagram | ||
|---|---|---|
|
| Code Block |
|---|
family acquisition
complete acq/data eq complete
family rt
complete data:ready or nrt eq complete
task wait
event data
cron 10:00 12:00 00:05
endfamily
task late
trigger not data:ready
time 11:00
family nrt
complete data:ready
task wait
event data
endfamily
task data
trigger rt/wait:data or nrt/wait:data
event ready
endfamily
|
...