The ability to execute Modules in parallel is a powerful feature offered by Composable DataOps Platform, benefiting DataFlow Application authors in various ways. By leveraging parallel execution, authors can significantly improve the overall execution time of their applications, particularly for long-running Modules that are computationally or input/output (I/O) intensive.
When a DataFlow Application is executed, Composable’s execution engine carefully analyzes the DataFlow structure to determine the optimal execution plan. It examines the dependencies between Modules and identifies opportunities for parallel execution. Modules that do not have any dependencies or whose inputs are readily available are scheduled for execution.
In Composable, a DataFlow Application is constructed using a collection of Modules, forming a directed acyclic graph (DAG). The DAG structure represents the dependencies between Modules, where the output of one Module serves as the input to another. When it comes to executing the DataFlow Application, the Composable Execution Engine employs a topological sort algorithm to determine the optimal execution plan.
The topological sort algorithm analyzes the DAG, ensuring that Modules are executed in the correct order based on their dependencies. It identifies the Modules that have no incoming dependencies, meaning their inputs are ready for processing. These Modules are scheduled for execution first. Once a module has been executed, its outputs are made available for the dependent Modules, and the process continues until all Modules have been executed.
By performing a topological sort, the Execution Engine maximizes efficiency and minimizes unnecessary waiting times. It ensures that Modules are executed only when their inputs are available, preventing any dependencies or data dependencies from causing delays in the overall execution. This approach optimizes resource utilization and allows for parallel execution when possible, resulting in improved performance and faster results.
The topological sort algorithm also helps in managing complex DataFlow Applications with multiple branches and interconnected Modules. It provides a clear execution order, ensuring that the data flows smoothly through the application, maintaining integrity and consistency throughout the process.
Next time you execute a DataFow, just click the ‘Parallel’ check box, and Composable will do the rest!
