Facilitating Development of Composable DataFlows in Code with Visual Studio

Composable’s DataFlow designer is a powerful tool for rapid development of data pipelines in an intuitive, visual setting. However, some users with programming experience may prefer developing DataFlows in code instead of visually, which is made possible by Composable’s FluentAPI.

The FluentAPI is a user-friendly way of creating, executing, and viewing the results of Composable data flows in code. It is a set of C# classes that interact with Composable Analytics services in order to provide a versatile and lightweight way of using data flows.

13.1 FluentAPI User Guide – Composable DataOps Platform

Currently, the FluentAPI supports DataFlow development in C#, Visual Basic, F#, and C++. For example, we could write the following C# snippet to create a DataFlow with a single calculator module that performs the operation 8 – 3:

This code is equivalent to adding the following calculator module to the designer canvas:

For more details and to see this C# example in its entirety, please visit the FluentAPI documentation.

Users can leverage the FluentAPI to build new DataFlows entirely in code, as shown in the above example, but they may also want to convert an existing DataFlow from its visual representation to a code representation. Thankfully, the Composable web app uses the FluentAPI to automatically generate source code for DataFlows in the designer, so you don’t have re-create your entire DataFlow! This is a useful time-saver, but unfortunately, it doesn’t completely overcome the high barrier to entry for DataFlow development in code.

To see why, let’s consider the local development setup process in more depth. What would the workflow be to continue developing an existing DataFlow locally in code, perhaps in an IDE like Visual Studio? To do this, you would have to:

1. Create a new Visual Studio console application with the proper .NET version.
2. Copy and paste the generated code from the designer into the main file of the Visual Studio project.
3. Set up some configuration files to make managing your Composable credentials and other relevant parameters easier and more secure.
4. Manually reference all of the required CompAnalytics DLLs so that the code can be compiled and run.

Once those steps are complete, you can begin developing the source code of your DataFlow. This setup process (especially step #4) can be somewhat tedious, even for simple DataFlows, and repeating it for multiple complex DataFlows would not be a scalable solution.

To make this process quicker and easier, Composable has introduced a Download Project button in the DataFlow designer that automates the above steps. Clicking this button will download a zip file containing a Visual Studio project with an auto-generated source code file in the selected language, the required DLLs, and any other supporting files necessary for local development. Now, you can edit, compile, and run the code right away after downloading, without needing to worry about setting up a new Visual Studio project, adding the proper dependencies, or completing any other setup steps.

Let’s demonstrate the download workflow with a small example. I’ve created a DataFlow in the designer that uses calculator modules to perform a simple arithmetic operation, in this case adding up the numbers from one to four by computing the parenthesized expression (1 + 2) + (3 + 4). This DataFlow is called “MyCalculator.” Here’s what it looks like:

To download the project for this DataFlow, click the </> Code button in the designer toolbar. Next, use the dropdown to select the language in which the DataFlow source code should be generated. The code viewer will change to reflect the updated language selection. Finally, click Download Project.

Unzip the download and open the solution file in the top-level directory (mine is called MyCalculator.sln), which will open our project in Visual Studio. Before trying to compile and run the code, open the App.config file and add your Composable username and password as values of the proper fields. This is a required step– the auto-generated code pulls credentials from this config file and will fail at runtime if you are not properly authenticated.

Finally, open the file containing the code–in a C# project, it will be called Program.cs–and click Start in Visual Studio to compile and execute the DataFlow code.

This completes the tutorial, but before I wrap up, I’d like to highlight one more convenient aspect of this new feature: the pre-compiled executable.

The downloaded zip folder includes a pre-compiled executable of the DataFlow source code. If you just want to be able to run the DataFlow without altering the source code, this executable makes it possible to do so without needing to open Visual Studio or invoke a compiler. To run the executable, navigate to the bin\Debug directory in the zip folder. You will need to update the config file in this folder (in my example, it’s called MyCalculator.exe.config) with your Composable credentials, as described above. This file is identical to App.config, just with a different name. Once your credentials are saved, you can double-click the executable (MyCalculator.exe) or run it using your shell of choice.

That’s all there is to it– you can now easily edit, compile, and run DataFlows in code using Visual Studio or the pre-compiled executable!