Sitemap

Reviving the messaging network for hybrid multi-cloud solutions

7 min readApr 24, 2025

--

Messaging continues to be essential in modern applications to enable decoupling of fine-grained cloud native componentry. However, what if the recipient of that message is in a completely different IT domain? How does the message get there and who is responsible for it whilst in transit? Sitting behind true enterprise messaging is a much more powerful capability that is all too often overlooked — a messaging network.

Only seeing messaging as queues and topics is a bit like only discussing mailboxes when talking about how the postal system works or only discussing power sockets when talking about household electricity.

The magic of the postal system is the underlying network that takes care of a package after you put it in the mailbox. It includes local sorting offices, global distribution by planes, ships, trains and mail vans, a common payment model, international agreements between different countries’ postal systems, and much more.

Similarly, electricity doesn’t just self-generate from the power sockets in your house. It is created by a network of power stations and renewable energy sources, collated and distributed across an electric grid spanning the country over pylons and underground cables. This is then transformed by substations into useable electricity that is finally connected into your home fuse box, and in turn, to your power sockets.

These short descriptions massively oversimplify the complexity behind the scenes, but you get the idea. For both the postal system and the electricity supply, the real magic is not in the part that you see (the mailbox or power socket) but the network behind it that you generally don’t need to be aware of.

We might only see queues and topics when communicating with a messaging system, but modern IT solutions require collaboration between applications across an ever-extending heterogeneous landscape.

New innovations typically tie together functionality from very separate parts of a business (different domains), each of which has significant autonomy over their technological choices. For example, what runtimes they use, what platforms they run on (containers, virtual machines, mainframes), and whether they run on-premises or in one of the major hyperscalers. There is a good chance that many of the parts of a modern solution will be distant from one another both physically and technologically. Even more so if your solution uses capabilities from business partners external to your organisation.

The go-to protocol for bridging domains is often HTTP based APIs, which certainly have their place and enable a simple programming model for many tasks. However, for more complex interactions, they can push a lot of complexity back into your application. API requests are dependent on the entire chain from caller to provider being available, and performant. To make an application robust to the very likely possibility of issues in a long chain between domains, you have to build in retry logic, additional persistence, idempotence, and maybe even reconciliation processes. You also have to consider the effect on performance and resources of your own application if network latency increases.

Messaging is specifically designed to decouple you from that chain of dependencies. Once an application has committed a message to a queue, it can move onto other work and be assured that the message will eventually arrive at the target application and be processed. As a result, the design and operation of the applications are significantly simplified. This is why in a world of solutions built on hybrid multi-cloud landscapes we’re seeing a resurgence of interest in event driven architectures and the asynchronous technologies that underpin them.

So, messaging has an important place in modern solutions, but to make that work, you need a lot more than just queues and topics to enable cross domain communication. You are going to need a messaging network that spans that hybrid cloud landscape.

Let’s explore some of the issues with cross domain transmission, and how a messaging network might help.

Latency: Time to traverse from one domain to another may be sufficiently high that it has a significant effect on the performance of the calling application. A messaging network will allow you to have a distributed set of messaging servers such that there is always one located close to your application for performant communication.

Network topology: There may be network boundaries such as firewalls to overcome between the two domains. A messaging network need only be set up once for all messaging interconnections and can provide advanced options to simplify firewall traversal. Granular access controls can then be applied at the messaging level as required.

Ingress/egress costs: Transmitting data to or from a cloud provider is often charged. A messaging network can ensure that the optimum path is taken and perform more efficient bulk compression than would be possible with direct connections.

Security models: The domains may use differing approaches to access and identity management. The network can enable access to queues based on localised security mechanisms.

The messaging network takes care of the complexities of reliable transmission of messages regardless of how separated the two applications are, on different networks in different sites, and from differing platform providers.

Let’s now look at some of the key properties of a messaging network from an architectural perspective.

Dynamic routing to relocatable endpoints: The actual location of the destination need not be known by the application. If the destination changes (e.g. during an application migration, or in a failure scenario), the network should be able to dynamically re-route without affecting the applications.

Lossless, exactly-once, efficient message transmission: Once the message is committed to the network, it should be moved atomically between network nodes until it is ready for collection at the destination. Transmission should never result in loss or duplicate messages, and can include additional efficiencies such as auto-batching and bulk compression.

Self-healing cross domain connections: Clients shouldn’t need to concern themselves with end to end connectivity, only communication with their nearest node. Connections between nodes in the network should be tolerant of momentary network issues, variations in latency, and should be able to route around ongoing failures.

Span locations, regions and cloud providers: The network should have mechanisms to span a variety of different types of network boundary. It should be possible to expand a network across data centers, into the cloud and across multiple cloud providers.

Multiple platforms and form factors: Network nodes should be able to run on suitable form factors for each part of the landscape. For example, virtual machines, container orchestration platforms, mainframes, dedicated appliances, and as a managed service. For each, it should be natively optimized for non-functionals such as performance, availability and security.

Supports relevant language interfaces and protocols: To send and receive messages, there should be client libraries/SDKs available for most popular languages, as well as the option to use a variety of standard protocols.

End to end security: It should be possible to ensure data privacy of messages throughout their journey, both in transmission and when they are at rest. There should also be granular access control to ensure that clients can only see specific queues and topics. The security models for identity and access might even be different on each end of the transmission.

If the messaging network can provide all of the above, that’s a whole lot of logic we don’t need to embed and maintain within our solution design. Our applications will have fewer dependencies, be more stateless, be easier to maintain and they will inherently be more robust.

These are key factors for any application, but especially when following cloud native principles and deploying onto modern platforms such as containers. Here the principles of decoupling are particularly important. Being able to efficiently communicate asynchronous messages with minimal surrounding logic makes for efficient code that is easy to write, and maintain.

The trick with the messaging network is that it can provide a point of presence within your application domain as a local resource that you own, but that resource is efficiently connected into the broader network. So, you can asynchronously connect to any connected remote application with the same confidence that you would to a local resource and the messaging network will take care of everything in between.

Full disclosure, I work for IBM and have had the great privilege of working more closely with the IBM MQ team recently.

What prompted this article was just how overlooked the well-established capabilities in IBM MQ around messaging networks were. With the resurgence in event driven architecture and at the same time, increasingly hybrid multi-cloud landscapes, it felt like the perfect time to bring messaging networks back to the surface and remind people how powerful they are.

--

--

Kim Clark
Kim Clark

Written by Kim Clark

Integration focused architect. Writes and presents regularly on integration architecture and design topics.

Responses (1)