Versioning Modules – Migration of publicly released interfaces

In this post, we will go over how Composable solves the problem with updating Modules, while avoiding breaking-changes to callers. Versioning is a tough problem. When code takes a dependency on your interface, and you need to update it … what do you do? In most object-oriented and functional languages, there are a few ways developers accomplish making changes.

• One option is to change the interface and force callers to update, or be broken. With internal code, this is the standard operating procedure. But this is a hard stance, and most people outside the inner circle of the code-base will be upset with breaking changes.
• If you simply need to accept another argument from the caller, there are a couple of options. One is to add an optional argument, and set it’s default value. Another way is to add an overloaded method with different arguments.
• If it’s a large change, then the supporting modifications typically involve adding an additional method with a different name, arguments, and return values. And you can then deprecate the old functions.  In a few versions down the road, most developers assume they can safely remove the code, and force any remaining callers of the old methods to update their code.
• The other option is to branch / snapshot the code base, and increment the version of your assemblies / packages. Callers will not be affected with changes to the newer versions of the assemblies, but won’t get any updates or features, unless developers back port non-breaking changes to the older branches.

Before jumping into how Composable versions Modules, let’s revisit the moving parts of Modules. Composable Modules have inputs and outputs (also called arguments). Developers specify their arguments through properties on a ModuleExecutor.  When a DataFlow Application references a Module, a snapshot of the current Module version is taken to allow for connections and values to hang off the inputs and outputs. Any meta attributes (control, description, validation) on the inputs and outputs will be referenced rather than copied. But what happens if you want to add a new argument, remove one, or change the type on an old one?

If you’re not taking advantage of the new upgrade features, then the following behavior will happen on update:

• If a developer removes an input or output, the existing inputs and outputs stay on old Modules (Modules created previously to deploying the change). Input values won’t be used, and output values won’t be set, so downstream modules won’t execute.
• If a developer adds an input or output, new Modules will get these inputs, but these will not show up on old Modules.
• If a developer modifies the argument type, and the new type is a more encompassing type (i.e string -> object), then it should be fine. Otherwise, this would be a breaking change.

There are a few issues with Composable just blindly adding or removing inputs and outputs for a user. For one, if we remove an input, and it had a crucial value associated with it (let’s say 10k line SQL query), then that would be bad value to loose. In addition, if we remove inputs and outputs that have connections, we are affecting the dependency order, and may break the flow and execution order. In addition, any downstream Modules that take inputs as connections, would now use their default inputs.  I could potentially see us adding new inputs in some scenarios, but since we’re not removing them, we might as well be symmetric.

Now, let’s discuss how a developer allows users to upgrade a Module to the latest version, while still maintaining backwards compatibility.

A ModuleType has a version, and a Module (the instantiated representation of a ModuleType) has a version.  A Module’s version number is the Module type’s version at instantiation time. A developer can increment the ModuleType’s version if argument changes are needed. These changes may include the following:

• Adding an input or output
• Removing an input or output
• Renaming an input or output (add / remove)
• Changing the type of a Module input or output (add / remove)
• Changing the order of inputs and outputs

Luckily, Composable provides a way to perform these changes, and many more.

To increment the Module type version, just set it in the ModuleType attribute. Version numbers start at 0 by default.

If you need to remove an argument, but still want old Modules to use it, and not be broken, you can use the [Obsolete] attribute.

Param2 can still be used in the Module’s execute method to support old behaviors and code paths, but Param2 won’t exist on any new Module instances

If you want to deprecate an entire Module, but still allow old instances to execute, you can mark the Module type class with the [Obsolete] attribute. Modules with this attribute will no longer show up in the pallette.

To perform an actual upgrade from one version to another, you need to create a Module upgrader. You can extend from the class ModuleUpgrader<T> where T: ModuleExecutor.

Each Module upgrader migrates a Module from one version to another version. You will typically have an upgrader for each version. An upgrader can essentially do anything they want to the Module to get into shape for the next version (i.e. adding arguments, removing arguments, changing / converting inputs values, and moving  or removing connections). The base ModuleUpgrader class does have several methods to make the ugprade job a little easier.

The default methods create the argument for you, and then it’s your job to add it to the Module in the right order, and move any values and connections over.

• ModuleInput CreateDefaultInput(string name)
• ModuleOutput CreateDefaultOutput(string name)

The migrate methods will create the new default default argument, move any connections and values to the new argument, and remove the old one. These may not work in some scenarios: sophisticated types, connections should be removed rather than moved, argument order matters.

• void MigrateInput(Contracts.Module module, string fromName, string toName)
• void MigrateOutput(Contracts.Module module, string fromName, string toName)