Composable DataOps Blog
Composable DataOps Blog
As organizations scale their usage of Databricks, they often encounter challenges related to governance and management of the various components within the environment. In this blog post, we will explore the importance of implementing a monitoring tool to maintain control…
Organizations are constantly seeking new ways to access and leverage valuable data. One emerging solution that has gained significant attention is the concept of data marketplaces. Data marketplaces provide a platform for securely sharing data, enabling organizations to tap into…
Composable DataOps Platform can be effectively used with Snowflake, a cloud-based data warehousing platform, to optimize and reduce the cost of Snowflake by leveraging Composable’s unique features and capabilities. Here are some ways in which Composable can help in cost…
The purpose of this Spotlight is to demonstrate the use of the DoWhile module to loop within a Composable DataFlow. Our scenario: The Fibonacci sequence is one of the best-known series in Mathematics. The series is composed of Fibonacci numbers,…
Composable has an SDK for deploying resources between two instances of Composable. For example, a source instance may be a development environment, and the target instance may be the production environment. Resources can include Dataflows, Queryviews, DataPortals, Keys, WebApps, etc…
Composable has an API for every action that you can perform through its UI. All services are documented here: Below is an example script for executing operations on the UserService. The first part of the script loads the necessary…
The purpose of this Spotlight is to demonstrate the mechanics of a simple, classic For loop within a Composable DataFlow. Our scenario: We are told to expect delivery of a plain-text dataset with the following column names: first name, last…
The WebService we’ll be creating is a simple API endpoint that accepts a Time Zone as a parameter and returns current date and time. To create a Web Service, first create a new DataFlow and add two modules: “Web Receive”…
DRY When building out any logical process, it is common to imagine reusability of components that you have built. Whether that is making a copy for yourself to take in a different direction, building out a template, or taking a…
Adding comments Composable DataFlows can span the range from simple tasks accomplished with a small number of easily-traceable modules to broad, complex, even nested tasks that become harder to follow the logic of as they evolve across dozens of modules…
Changing Wire Type Some people like the default curvy connectors between the modules in a Composable DataFlow, but some would like a different look. Did you know that those connectors are changeable? As an example, let’s explore the Composable DataFlow…
The purpose of this Spotlight is to explore how one could compare modules and evaluate which to use within a Composable DataFlow. This Spotlight illuminates the first lab of the Project Lab Series: NASDAQ API. It picks up from the…
Composable’s DataFlow designer is a powerful tool for rapid development of data pipelines in an intuitive, visual setting. However, some users with programming experience may prefer developing DataFlows in code instead of visually, which is made possible by Composable’s FluentAPI.…
Tutorial “Effective October 1, 2022, we will begin to permanently disable Basic Auth in all tenants, regardless of usage, with the exception of SMTP Auth.” Due to the recent deprecation of both Exchange Online and support for Basic Authentication,…
What is a Predicate? A predicate is an operation on an Input object which produces a boolean (true/false) result. It can be helpful to think of a predicates as “Yes or No” questions; “Is this car red?” “Is this building…
Have you ever received the error System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.? If so,…
In this post, we’ll be going over how to configure an ODBC connection to DataBricks within Composable so you can run spark queries from Composable resources (i.e. QueryViews and DataFlows). The first thing you’ll need to do is install the…
For a previous task, I was reliant on external API’s to work correctly and to know when functionality had changed. With proper acceptance/validation testing on foreign API’s, you can sleep a little bit better at night knowing that the behavior…
This post will cover two recent additions to Composable DataFlows that have yet to receive a comprehensive write-up: DataFlow Resume and Pinned Results. We’ll be discussing them together due to their conceptual overlap; both facilitate DataFlow development and debugging by…
Many first time users of Composable have asked “How does Composable compare to Streamsets?” I finally had the opportunity to use Streamsets in a real-life setting a couple of months ago, and boy am I glad I did. Hopefully this…