What is ecFlow
ecFlow is a client/server workflow package that enables users to run a large number of programs (with dependencies on each other and on time) in a controlled environment. It provides reasonable tolerance for hardware and software failures, combined with restart capabilities.
ecFlow submits tasks (jobs) and receives acknowledgments from the tasks when they change status and when they send events, using child commands embedded in your scripts. ecFlow stores the relationships between tasks, and is able to submit tasks dependant on triggers, such as when a given task changes its status, for example when it finishes. Users communicate with ecFlow server using:
- Command Level Interface (CLI).
- Python interface
- GUI
ecflow_ui. (QT based GUI)
ecflowview. This is an X-Windows/Motif based program. (This will be deprecated in the future)
ecFlow runs as a server receiving requests from clients. CLI, GUI and the suite jobs are the clients. Communication is based on TCP/IP. Note that ecFlow is a scheduler and is not a queuing system such as NQS, SGE, Load leveller or PBS. However, it can submit to queuing systems.
History
ecFlow is a product of the European Centre for Medium-range Weather Forecasts (ECMWF) and was written by the Development section.
ecFlow is a complete replacement of SMS. It has been written using an object oriented language. It provides most of the functionality that is available in SMS.
Differences with SMS
- Maintenance and enhancement of server software easier.
- SMS provided a custom scripting language for defining suites. In ecFlow you have the choice of using either a simple text based format or a Python API as a replacement for CDP/text interface. The entire suite definition structure can be specified in python.
- Any language can be used to create the textual suite definition file as it has a published format. This is different to current SMS where suite definition keywords are treated as commands.
- Allows better error checking and handling of zombies.
- Will work on 64 bit operating systems like AIX without the need to compile in 32 bit mode.
- Does not require an explicit login.
ecFlow excecutables
ecFlow functionality is provided by the following executables and shared libraries
- ecflow_client: This executable is a command line program; it is used for all communication with the server. This executable implements the Command Level Interface (CLI). The bulk of this functionality is also provided by the python API
- ecflowview: This is a specialised GUI client that monitors and visualises the node tree hierarchy.
- ecflow_ui: This is a specialised GUI client that monitors and visualises the node tree hierarchy. Based on QT.
- ecflow_server: This executable is the server. It is responsible for scheduling the jobs and responding to ecflow_client requests.
- ecflow.so: These shared libraries provide the python API for creating the suite definition and communication with the server.
Terminology and typography
Command names in this text are written in this fashion, and corresponding sections in the manual pages is shown within parentheses. UNIX manual page sections are numbered from one to eight; for example, the UNIX C shell is shown as ksh(1) which indicates it is in the first section of the UNIX manual pages. load (CLI) indicates the command load is a CLI command(i.e. implemented by the ecflow_client executable)
Through the rest of this document, 'ECF' will be used interchangeably with ecFlow. Any references to server, client or GUI, refers to ecFlow server, client and GUI
A concept is a name used in this document to describe an idea. A variable is used a lot throughout this document. A node represents a task, family, suite or equivalent when it does not matter which we are talking about.
Extracts from script or definition files look like this # This is in a file and this is important in this file some other lines
Colours are also used to highlight the status of a node, so if node /x/y/z is active, it would look like /x/y/z.