Composable has recently added progress-tracking features, including progress metrics and animations. Both of these aim to elucidated DataFlow execution by showing the modules that are currently executing and highlighting potential bottlenecks.
When executing a DataFlow, users will now see blue circles animating along connection paths, feeding into the inputs of executing modules. Also, users no longer have to wait for the entire DataFlow to finish executing to see module outputs; once a module is complete, its outputs will be filled in mid-execution. This gives users to ability to monitor any intermediate results and diagnose bugs as soon as possible.
The progress metrics give users a sense of execution speed and statistics at the module level. These are stored in the progress table, which can be found as a tab in the trace log, titled “Progress”. This table includes basic execution information for each module: start time, end time, duration, iteration count for loops.
The table also provides overall statistics, such as total duration. Streaming modules will have extra statistics here, including average streaming rate, and total rows streamed.
Additionally, the table shows individual iteration metrics for streaming modules (a row per streaming batch) and looping modules (a row per loop). These allow users to explore bottlenecks not only at a module level, but at an iteration level as well. For example, users can see that a module took twice as long on loop 2 as on loop 3.
These metrics can also be found by right-clicking on a module and exposing its context menu. The menu gives options to view statistics and iteration information for a given module.
This update also includes minor improvements to how Composable shows past runs and how users can explore old traces. When opening an old run in the Designer, users will now have access to all available traces and metrics from that run. For traces, we’ve added the option to toggle between the latest view (showing the last 10,000 traces), and a paged view. While the latest view is often sufficient, DataFlows will hide their first sets of traces (and potentially important exceptions) if the trace count exceeds 10,000. The paged view is a natural solution to this problem, allowing users to view all traces regardless of trace count.
We hope these features will improve your DataFlow-building experience in Composable. Take a look at these progress metrics next time you’re debugging a DataFlow.
