ecFlow's documentation is now on readthedocs!

Please go to https://ecflow.readthedocs.io/en/latest/index.html

SMS triggers are checked from the client side. It is simple to reach a situation where triggers are not valid from the server side, and trigger related task may remain queued (deadlock).

To prevent that to happen, ecFlow triggers check is more robust:

  • triggers must be consistent from the client side.
  • ‘extern’ shall only refer to an external suite’s node
  • server may be requested to confirm that all triggers are referring to an existing node, in already loaded suites.

As it is with the SMS, a ecFlow suite can be shared for team work. It is suite design and management skill to decide how to split the suite for a good collaboration within team members.

At ECMWF, operational suites are defined in files describing the full content. * the definition file is split into different python modules for

  • observations retrieval
  • analysis and forecast
  • products generation and dissemination
  • plots generation
  • then a python script is in charge to put all things together and to create the suite
  • the user decides to load the full suite, only a subt-ree from the suite or to load it as an e-suite (test suite)
  • a SCM is used to identify the modules edited by different users, merge their updates, resolve the conflicts or allow them to work in an branched version or the suite for the next release.</li></ul>

The following example shows that with ecFlow

  • a master definition file may provide the suite structure (multi1.def), with the right setting of inlimits, limits, triggers,
  • a developer in charge of adding test tasks to the suite may update it with a task test2, below an agreed family (/multi/main/00/test),
  • another tester may add some other tests as a different suite, triggered from ‘multi’ suite.


  ecflow_client --load multi1.def
  ecflow_client --replace /multi/main/00/test/fc_complete/test2 multi2.def
  ecflow_client --replace /test20121123 multi3.def

  python TestBench.py multi1.def
  python TestBench.py  multi3.def

multi1.def: suite structure definition


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
suite limits
  defstatus complete
 limit power6 120
  limit power7 120
  limit prod   10
  limit diss   10
  limit test 10
endsuite

suite multi

  family main
    inlimit /limits:power6
    repeat date YMD 20120101 20301212

  family 00
    family obs; 
      task get; 
    endfamily
    
    family ana
      trigger obs eq complete
 task ifs
    endfamily

    family fc
 trigger ana eq complete
 task model
        meter step -1 120
    endfamily

    family prod
    inlimit /limits:prod
      family 000; 
    trigger /multi/main/00/fc/model:step gt  0 or /multi/main/00/fc/model eq complete
 task gen; endfamily
      family 012;
    trigger /multi/main/00/fc/model:step gt 12 or /multi/main/00/fc/model eq complete
 task gen; endfamily
      family 024; 
    trigger /multi/main/00/fc/model:step gt 24 or /multi/main/00/fc/model eq complete
 task gen; endfamily
      family 048;
    trigger /multi/main/00/fc/model:step gt 48 or /multi/main/00/fc/model eq complete
 task gen; endfamily
      family 096; 
    trigger /multi/main/00/fc/model:step gt 96 or /multi/main/00/fc/model eq complete
 task gen; endfamily
      family 120; 
    trigger /multi/main/00/fc/model:step gt 120 or /multi/main/00/fc/model eq complete
 task gen; endfamily
    endfamily prod

    family diss
    inlimit /limits:diss
      family 000; 
    trigger /multi/main/00/prod/000 eq complete
 task gen; endfamily
      family 012;
    trigger /multi/main/00/prod/012 eq complete
 task gen; endfamily
      family 024; 
    trigger /multi/main/00/prod/024 eq complete
 task gen; endfamily
      family 048;
    trigger /multi/main/00/prod/048 eq complete
 task gen; endfamily
      family 096; 
    trigger /multi/main/00/prod/096 eq complete
 task gen; endfamily
      family 120; 
    trigger /multi/main/00/prod/120 eq complete
 task gen; endfamily
    endfamily diss

    family test
 inlimit /limits:test

      family fc_complete
    trigger /multi/main/00/fc eq complete
 task test1
      endfamily

      family prod_complete
    trigger /multi/main/00/prod eq complete
 task test1
      endfamily

      family diss
    trigger /multi/main/00/diss eq complete
 task test1
      endfamily
    endfamily

multi2.def: task and family addition for intra-suite test


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
extern /limits:test
suite multi
  defstatus suspended
  family main
  family 00
  family test
 family fc_complete
    task test2

    family fam_test
      trigger test2 eq complete
 task prepare

      task process
        trigger prepare eq complete

 task push
        trigger process eq complete

multi3.def: standalone test triggered by ‘multi’ suite

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
extern /limits:test
extern /multi/main/00/fc/model:step
extern /multi/main:YMD

suite test20121123
  defstatus suspended # so that it can be changed manually to queued
  repeat date YMD 20120101 20301212
  family test
 repeat integer STEP 0 120
    trigger /multi/main:YMD gt /test20121123:YMD or (/multi/main:YMD eq /test20121123:YMD and (/multi/main/00/fc/model:step ge /test20121123/test:STEP or /multi/main/00/fc/model eq complete))
    task run