web analytics

Configuring MSMQ with Windows Failover Clustering on AWS for Composable DataOps (2-Node Setup)

Microsoft Message Queuing (MSMQ) is a robust messaging platform for reliable and low latency asynchronous communication between distributed application components. It offers exactly-once delivery, transactional coordination with databases, priority queues, and peek-lock semantics — making it well-suited for mission-critical workloads that demand durability, ordering, and seamless failover.

By combining MSMQ with Windows Failover Clustering (WSFC) and shared storage (EBS Multi-Attach or FSx SMB), you can achieve a highly available MSMQ deployment on AWS EC2 for the Composable DataOps Platform. if one MSMQ node fails, the other takes over automatically, allowing Composable to continue to operate seamlessly. Composable will connect to the queue via a single cluster name/IP.

Once MSMQ is configured in a cluster, the only required change within Composable is to point it to the cluster rather than a local machine MSMQ instance.

For example, you probably have the following AlertQueuePath setting in your in Web.config and CompAnalytics.ActivationService.exe.config config files:

<add key="AlertQueuePath" value="FormatName:Direct=OS:machinenamexyz\private$\alert-activator-queue" />

This would change to

<add key="AlertQueuePath" value="FormatName:Direct=OS:clustername\private$\alert-activator-queue" />

Why MSMQ Fits High-Availability Architectures like Composable DataOps

Continuous Operation with Local Queues – Producers keep sending even if the central queue is offline; messages are stored locally and forwarded later.

Transactional Messaging and Database Coordination – Exactly-once delivery with atomic commits across queues and databases.

Safe Message Handling (Peek-Lock) – Lock messages for inspection and rollback safely without removal.

Priority Queuing – Assign priority levels so urgent messages bypass normal traffic.

Strict Ordering and Durability – Transactional queues maintain order and persist through failovers.

AWS HA Alignment – Combines cleanly with EBS Multi-Attach or FSx SMB for seamless failover.

Architecture Overview

Components:

  • Two Windows EC2 instances (Nodes 1 and 2) in the same or different VPC/AZ
  • Shared storage (EBS Multi-Attach or FSx SMB) for MSMQ data
  • Cluster Name/IP as the client endpoint
  • Quorum witness (File Share or Cloud Witness) for split-brain protection

High-Level Flow:

  1. Both EC2 instances join an Active Directory domain.
  2. Failover Cluster is created with both nodes and shared disk.
  3. MSMQ is installed and configured to use shared storage.
  4. The cluster manages MSMQ as a Generic Service role with automatic failover.

Step-by-Step Configuration

1. Prepare AWS Environment
  • Launch two Windows Server EC2 instances (2019 or 2022) in the same Availability Zone if using MultiAttach EBS, or different Availability Zones if using FSx..
  • Join both instances to the same Active Directory domain.
  • Attach a Multi-Attach EBS volume (io1/io2) or configure FSx SMB share for queue storage.
  • Ensure security group rules allow:
Install Required Features

Run this on both nodes to install MSMQ and clustering features:

Add-WindowsFeature MSMQ-Server
Add-WindowsFeature Failover-Clustering -IncludeManagementTools

Verify installation:

Get-WindowsFeature MSMQ-Server, Failover-Clustering
Validate Cluster Configuration

Run cluster validation to ensure networking, storage, and configuration are correct:

Test-Cluster –Node NODE1,NODE2

4. Configure Shared Storage

EBS Multi-Attach
  1. Attach the volume to both EC2 nodes.
  2. On Node 1:
  3. Take the disk offline on Node 1 so the cluster can manage it.
Get-Disk | Where-Object PartitionStyle -Eq 'RAW' | Initialize-Disk -PartitionStyle GPT
New-Partition -DiskNumber 1 -UseMaximumSize -DriveLetter Q
Format-Volume -DriveLetter Q -FileSystem NTFS -NewFileSystemLabel "MSMQData"
FSx SMB
  • Create a folder (e.g., \\fsx-server\msmqshare) to store MSMQ data.
  • Ensure both nodes have Full Control permissions on the share.

5. Create the Cluster

Run this on one node to create the cluster:

New-Cluster -Name MSMQCluster -Node NODE1,NODE2 -StaticAddress <ClusterIP>

Verify cluster creation:

Get-Cluster

Add shared disk to the cluster (if EBS):

Get-ClusterAvailableDisk | Add-ClusterDisk

6. Configure Quorum Witness

For two-node clusters, configure a witness:

File Share Witness:

Set-ClusterQuorum -FileShareWitness \\fileserver\witness

Cloud Witness (Azure):

Set-ClusterQuorum -CloudWitness -AccountName <storageaccount> -AccessKey <key>

7. Configure MSMQ Storage Path

MSMQ stores data in %windir%\System32\msmq\storage by default. Change it to use the shared disk or SMB path:

$storagePath = "Q:\msmq\storage"
New-Item -ItemType Directory -Force -Path $storagePath
reg add "HKLM\SOFTWARE\Microsoft\MSMQ\Parameters" /v StorePersistentPath /t REG_SZ /d $storagePath /f
Restart-Service MSMQ

Repeat on both nodes.

8. Create MSMQ Cluster Role

Use Failover Cluster Manager or PowerShell:

Add-ClusterGenericServiceRole -ServiceName MSMQ -Name MSMQRole -StaticAddress <ClusterIP> -Storage "Cluster Disk 1"

9. Test Failover

Move the role manually to confirm:

Move-ClusterGroup -Name "MSMQRole" -Node NODE2

Check:

  • Clients can still access queues using the cluster name/IP
  • Messages remain intact after failover

AWS-Specific Considerations

  • Same AZ Requirement: Multi-Attach EBS works only in a single AZ. For cross-AZ, use FSx SMB.
  • Elastic IP: Failover Cluster IP cannot directly use an Elastic IP; use Route 53 health checks or automation to reassign EIP.
  • Instance Types: Multi-Attach EBS requires Nitro-based instances (e.g., M5, C5, R5).

Conclusion

This configuration enables Composable DataOps to run MSMQ in a highly available topology on AWS:

  • Single endpoint for producers and consumers (cluster name/IP)
  • Exactly-once delivery and priority queue handling
  • No message loss during failover events
  • Leverages AWS-native storage (EBS Multi-Attach or FSx SMB) with Windows clustering

Related Posts