ecFlow is a work flow package that enables users to run a large number of programs (with dependencies on each other and on time) in a controlled environment. It provides tolerance for hardware and software failures, combined with good restart capabilities. It is used at ECMWF to run all our operational suites across a range of platforms.
ecFlow submits tasks (jobs) and receives acknowledgements from tasks when they change status and when they send events. It does this using using child commands embedded in the scripts. ecFlow stores the relationship between tasks and is able to submit tasks dependent on triggers.
History
For almost three decades ECMWF used SMS (Supervisor Monitoring Scheduler) package to control the work flow for the Centre's operational models and systems. Written at ECMWF, it allows the design, submission and monitoring of jobs both in the Research and Operations Departments, and provides common tools for scientists, analysts and operators to cooperate. A large number of organisations also use SMS (from both Member and non-Member States). Development and support of SMS has now stopped.
The replacement, ecFlow 4, has superseded SMS. It has been used for over decade. It is a complete rewrite using object oriented methodology and modern standardised components; it acts as a comprehensive replacement for SMS. The rewrite helps improve maintainability, allows easier modification and introduces object orientated features. The proprietary script language used by SMS, CDP, have been replaced by Python.
Development of ecflow 4 has now stopped. But support is still provided. It has been replaced by ecflow 5
You can find an introduction to ecFlow in the ECMWF Newsletter article (Autumn 2011), starting at page 30.