One of the nice things about GigaSpaces is that heterogeneous independent applications can make use of GigaSpaces as a high performance fault-tolerant messaging hub. It has support for JMS in which applications can create topics and queues for exchanging messages. In this post, I would like to describe a design pattern that I have used for the design of a GigaSpaces Messaging for Apama.
In one of my current projects, I need to design a high performance, highly-available and scalable market data system (MDS) which is used by many different types of trading systems in a bank. We chose GigaSpaces as the middleware to build the MDS due to its simplicity; it provides messaging capability, high availability, scalability and performance in one single platform which simplifies a lot of development overhead, deployment issues and maintenance efforts. One of the requirements of the MDS is to integrate with Progress Apama. Apama provides a robust integration framework which facilitates the integration with many different event sources into Apama. Apama offer many popular connections out-of-the-box such as JMS, TIBCO Rendezvous, just to name a few. I decide to use JMS as the messaging transport to stream data from GigaSpaces to Apama. This requires us to develop a GigaSpaces JMS channel adapter for Apama.
GigaSpaces channel adapter
The GigaSpaces JMS channel adapter acts as a messaging client to the MDS and invokes Appama functions via the supplied interface. The adapter in this case is unidirectional since data is only streamed in one direction (from GigaSpaces MDS to Apama). GigaSpaces JMS already provides all the necessary mechanics to build the channel adapter so that Apama can interact with it similar to a JMS provider. Before the message can be delivered to Apama, we need to convert the market data format into Apama specific message. Therefore, a message translator is designed. This way, we decouple the adapter from the Apama internal structure.
Another implementation consideration of the channel adapter design is about transaction semantics. Since the message is streamed from the MDS to the message channel specific for Apama, we need to ensure that the message is delivered to Apama even during the event of partial failure. GigaSpaces JMS provides transactions so that message consumers can receive messages transactionally through the JMS session. We decide to stick with local transactions because it is more efficient. Hence, we will need to design a message channel that receives all messages that are relevant to Apama from GigaSpaces MDS.
GigaSpaces Message Channel
Once we have the GigaSpaces JMS channel adapter ready, we need to build a message channel where the channel adapter can consume market data from. The market data are fed into a GigaSpaces IMDG with a primary backup partitioning topology. This means that the market data are spread across multiple machines. The market data are partitioned based on the stock symbol (Content-Based Routing). This topology is essential for a high performance load-balanced MDS. Unfortunately, we can't use the market data IMDG directly as the message channel for Apama. As noted in the GigaSpaces documentation, GigaSpaces JMS does not support destination partitioning. Currently, all messages of a JMS destination are routed to the same partition. The partition where the messages go is resolved according to the index field of the message which, currently, is the destination name. In other words, we can't use the market data IMDG directly as the message channel for Apama because the market data are partitioned across multiple machines. In order to allow Apama to consume JMS message, we create another space which is uniquely designed for Apama (Apama Space Message Channel). This space will store only JMS market data messages that Apama is interested. In order to reduce the number of network hops for delivering market data to Apama, the Apama space message channel is built-in to the GigaSpaces channel adapter. To deliver a market data of interest from the Market Data IMDG to the Apama space message channel, a messaging bridge is designed. Basically, When a message is delivered on the market data IMDG, the bridge consumes the message of interest and sends another with the same contents on the Apama space message channel. The notify event container is implemented in the bridge to capture all updates for the market data of interest and the market data POJO message is transformed into a JMS message before it is written into the Apama space message channel. Note that in order to guarantee that the notification of a new update is received, the Market data IMDG will send the notification at least once in case of failover. This is fine for us as the trading algorithm is idempotent to the duplicate market data message.
With the current design, new applications which are interested in the market data messages can tab into the MDS in similar manner without affecting other existing applications. Applications are loosely-coupled by using GigaSpaces as a high performance message hub.
