The purpose of this Spotlight is to explore how one could compare modules and evaluate which to use within a Composable DataFlow.
This Spotlight illuminates the first lab of the Project Lab Series: NASDAQ API. It picks up from the second task of Connect to the NASDAQ API. It is written from the perspective of that lab, but it is not necessary to have done that specific lab work before following the steps in this Spotlight.
For context, in that lab, we had just created a new DataFlow and filtered the modules down by searching on “web”. If you are starting this Spotlight without having done that lab, you can create a new DataFlow, search on “web”, and be ready to go.
Let’s begin…
Looking at the filtered results under WebClient Data Sources (under Data Sources), we see a number of options that work with files, which is not relevant to use for our API. There is also an option for WebMultipartFormData, which is for form creation and file output. Handy in other use cases, but not what we need to connect to our API.
That leaves us with two simple options, WebClient and WebClient Robust. What’s the difference? Well, let’s investigate.
We can start our comparison by dragging a WebClient and then a WebClient Robust onto our DataFlow.
A quick side-by-side visual comparison shows the differences between the standard WebClient (“Standard”) module and WebClient Robust (“Robust”). For inputs (green dots on the left), Robust adds two authenticating properties to the bottom of the stack. For outputs (blue dots on the right), Robust has four, where Standard has only one. We can see more detail about the differences by using the built-in module help, accessed by clicking the ? icon at the top of any module.
From the details of the Standard module, we can see that its purpose is simply to retrieve the targeted contents as a string.
From the details of the Robust module, we can see that it will allow you to not only pass credentials as input, but also return status information and web response headers to go along with the response string returned as with Standard.
We will eventually want to choose only one of these for our API connector. To help with that choice, let’s explore how each handles a simpler API connection – both with success and with failure.
One fun API connection that doesn’t require any credentials is: https://catfact.ninja/fact. To no one’s surprise, this returns a random fact about cats each time you run it.
To test each module’s behavior, put that connection string into the Uri input for both the Standard and Robust modules. Since we are using the API for reading, not writing, we’ll also want to change both Method inputs from POST to GET.
That’s it. Now just click the green Run button at the top of the DataFlow.
You may notice the modules’ icons (upper left) quickly change to spinning cycles to show that they are active, then return to their original state. It’s a pretty simple process, so it doesn’t take very long.
So, what did we end up with?
If you right-click on the single output from the Standard module’s upper right, you see a pop-up context menu that shows View Results, the datatype returned (in this case, String), and some suggestions for modules that might continue our DataFlow.
Let’s click View Results.
You should get a pop-up window that contains, in this case, JSON data with two Key:Value pairs, for fact and length. As desired, you can use the icons in the upper right of this window to copy (the content), maximize/restore, and close. There is a scrollbar at the bottom if your fact is longer than can be displayed in the window. The window is also resizable if you want a wider window to read more text at a time.
If we follow the same steps for each of the four outputs from Robust, we see something like the following:
Yes, we can see all of the results at once and move them around! These windows are not Modal (locking focus until they are addressed).
In addition to getting another random cat fact in the Result window, the statuses of 200 and Success tell us that the process worked, and the ResultHeaders would let one dive into metadata properties of the web response.
OK. We have seen what happens when these modules get the desired result. How would they handle failure?
Let’s switch the Method of each module back to POST. Since we don’t have the ability to write a new cat fact with this API, that should generate a problem.
When we click Run again, this time, we get an error on the Standard module and a process break in our run. We can see that neither module finished, since the outputs are all grey instead of blue.
Clicking on the red exclamation point shows us the error message.
In order to see how Robust would handle an error, we have to get past Standard. We could just change POST back to GET, but we could also disable the Standard module to prevent it from running. To do that, we can right-click on the Standard module and select Disable.
Now, our Standard module has a red background. That indicates that it is Disabled and will not be active when the DataFlow runs.
When we click Run again, we now see that there is no error for Standard (it didn’t run, so couldn’t throw an error) and Robust has completed.
However, a quick look at the statuses and the Result shows that we did, in fact, have an error. It’s just that it was trapped and documented, and the process finished to the end.
(Note that in both error cases, there is also a notification in the Trace Log at the bottom of the DataFlow, so you don’t have to manually check for failure for each run.)
Given that the NASDAQ API may be more complicated than pulling random cat facts, I’m going to lean in favor of using the Robust module to give us more information if we run into an error.
So, for those heading back to the lab that launched this Spotlight, you can delete the Standard module (using the X in the upper right, or by selecting it and using the keyboard). The next step in the lab is going to adjust the Robust module, so you can configure the Robust module as part of that process.
That’s the end of this exploration into comparing and evaluating modules. You might have the opportunity to do a similar comparison and analysis any time you have multiple modules to choose from.
In our case, we can also return to the main Project Lab Series lab that sent us here and pick up where we left off in our journey to Connect to the NASDAQ API.
Enjoy!
