We often think of Composable as an IDE for developing data-intensive applications, and we, therefore, strive to provide a full development experience within the Composable web UI. To further that goal, we added global DataFlow level error handling options that allow Composable DataFlow developers to decide how to react to errors.
There are three options that encompass error handling at the global* (DataFlow-level):
- Continue On Error
- Email On Error
- Execute On Error.
(* Note that Module-level error handling is also available, and can be configured on each Module by clicking on the Module’s gear icon.)
These settings can be configured by clicking on the Error Handling Options button from the Properties Panel on the right side of the Designer.
Clicking on the Error Handling Options button pops up the options modal:
Continue On Error is exactly what it sounds like- if an error occurs during a Dataflow, it is logged, but the Dataflow will continue. The setting can be set at either the Dataflow level or the Module level, allowing for further precision within the Dataflow as the Module level setting will override the setting at the Dataflow level. It does not however, have to be set on every Module- if a Module does not have it set, it defaults to whatever the Dataflow level setting is at the time. When using Continue On Error, it’s important to note that any failed Module’s outputs are erased, and will not send data to a downstream Module. This is because that data is not necessarily trustworthy after a failure, so to prevent any inconsistent results, the output values are removed. With the exception of the Optional List Builder, Alert Sender, and the Consolidator, any Module directly connected to the failed Module will not execute because the input that connects to the failed Module will never be satisfied. It is important to keep this in mind while building a Dataflow so that Modules have the correct behavior after a failure. Every error that occurs is written to the trace and can be seen there when reviewing the DataFlow Run. The Run will have a status of ‘Success’ because no error was allowed to fully surface during execution, and as a result, any further error handling options will not be activated.
In order to have error suppression work in tandem with the other options, set Error On Completion to true in the settings modal. Note that it will not do anything unless Continue On Error is set somewhere in the Dataflow, because errors are logged at immediate failure normally. When set to true, all of the errors are collected in the background and then stored until the end of the Dataflow execution, where they are appropriately logged. The same rules for Module failures apply with Error On Completion set as with Continue On Error- all the output values will be removed from the offending Module. In addition to these suppression and logging settings, we also have reflexive actions that can be taken if a failure occurs.
Email On Error and Execute On Error allow for notification and immediate action upon an error occurring. Email On Error emails a person or persons when an error occurs. When set, it defaults to emailing the Dataflow Owner, but that can be overwritten by a comma-delimited list of emails should it be necessary. Execute On Error allows the user to specify another Dataflow if the one running fails. In order to provide information about what occurred, Composable passes error and run information to the reacting Dataflow so that actions can be taken based on the type or error logged or what Module failed previously. Re-running tasks, updating external databases post error, correcting any potential partial Dataflow completion results, and more are all possible.
With the additions of these types of error manipulation, we are excited to see the ways that Composable developers can be further empowered to automate their processes.
