Composable DataOps Blog
Composable DataOps Blog
Microsoft Message Queuing (MSMQ) is a robust messaging platform for reliable and low latency asynchronous communication between distributed application components. It offers exactly-once delivery, transactional coordination with databases, priority queues, and peek-lock semantics — making it well-suited for mission-critical workloads…
In order to meet the demand of incoming requests to a Composable cluster, you many need to scale out and run multiple web nodes. In this topology, a network load balancer is required to sit in front of the web…
There are many instances when working with DataFlows where you need to execute some executable or command. In this post we will explore two examples where the process executor was used to run a program not native to Composable from…
1. Create an App Registration in the Azure Portal. 2. In the App Registration, you’ll need to add Composable redirect urls. There are 2 different urls that Composable may request Microsoft Online to redirect back to. One is the Composable…
As we step into 2025, the landscape of artificial intelligence (AI) and DataOps continues to evolve at an unprecedented pace. At Composable, we’ve been at the forefront of these advancements, and we’re excited to share some recent industry recognition and…
As we close out 2024, we wanted to share some of the latest updates and enhancements to the Composable platform. These features, ranging from major upgrades to subtle yet powerful enhancements, are designed to streamline your processes and supercharge your…
Composable now supports running Python scripts from DataLabs using the DataLab Notebook Runner Module. This is useful in production environments where you wish to run DataLab code without the overhead of spinning up a notebook. The above script and notebook…
ChatGPT is a revolutionary language model that has taken the world by storm. With its remarkable ability to generate human-like responses and engage in meaningful conversations, it has become an indispensable tool for various applications. The benefits of ChatGPT are…
As organizations scale their usage of Databricks, they often encounter challenges related to governance and management of the various components within the environment. In this blog post, we will explore the importance of implementing a monitoring tool to maintain control…
Organizations are constantly seeking new ways to access and leverage valuable data. One emerging solution that has gained significant attention is the concept of data marketplaces. Data marketplaces provide a platform for securely sharing data, enabling organizations to tap into…
Composable DataOps Platform can be effectively used with Snowflake, a cloud-based data warehousing platform, to optimize and reduce the cost of Snowflake by leveraging Composable’s unique features and capabilities. Here are some ways in which Composable can help in cost…
The purpose of this Spotlight is to demonstrate the use of the DoWhile module to loop within a Composable DataFlow. Our scenario: The Fibonacci sequence is one of the best-known series in Mathematics. The series is composed of Fibonacci numbers,…
Composable has an SDK for deploying resources between two instances of Composable. For example, a source instance may be a development environment and the target instance may be the production environment. Resources can include Dataflows, QueryViews, DataPortals, Keys, WebApps, etc.…
Composable has an API for every action that you can perform through its UI. All services are documented here: Below is an example script for executing operations on the UserService. The first part of the script loads the necessary…
The purpose of this Spotlight is to demonstrate the mechanics of a simple, classic For loop within a Composable DataFlow. Our scenario: We are told to expect delivery of a plain-text dataset with the following column names: first name, last…
The WebService we’ll be creating is a simple API endpoint that accepts a Time Zone as a parameter and returns current date and time. To create a Web Service, first create a new DataFlow and add two Modules: “Web Receive”…
DRY When building out any logical process, it is common to imagine reusability of components that you have built. Whether that is making a copy for yourself to take in a different direction, building out a template, or taking a…
Adding comments Composable DataFlows can span the range from simple tasks accomplished with a small number of easily-traceable modules to broad, complex, even nested tasks that become harder to follow the logic of as they evolve across dozens of modules…
Changing Wire Type Some people like the default curvy connectors between the modules in a Composable DataFlow, but some would like a different look. Did you know that those connectors are changeable? As an example, let’s explore the Composable DataFlow…
The purpose of this Spotlight is to explore how one could compare modules and evaluate which to use within a Composable DataFlow. This Spotlight illuminates the first lab of the Project Lab Series: NASDAQ API. It picks up from the…