Two Strategies For Asynchronous Message Processing

Asynchronous Message Processing

Asynchronous message processing is a common pattern in designing highly available web applications. Essentially, it means that certain tasks are not processed immediately within the scope of a HTTP request. Instead, they are handled asynchronously, by a different application unit, which in Microsoft realm is often (but not always) implemented as a Windows service. This helps streamline the web application and make it appear faster to the end user.

There is a special class of tasks that are suitable for asynchronous message processing, and it is different from typical scheduled jobs (also executed by a separate application service). Usually, they are closely related to user interactions with the main web application, but are too process-expensive or slow to be incorporated into the page processing workflow. Dynamic rendering of a multi-page PDF report is a good example since it falls under both categories. Sometimes, the tasks are not necessarily slow, but they do not produce any information for the user, for example keeping an audit trail of user activity within the system.

Regardless of the specific task being "outsourced", web application exchanges information with the processing service using discrete messages. In a typical design, messages flow only one way: from web application to the service (although there are scenarios where two-way dialog is preferable). There are two main strategies for implementing this design, MSMQ and SQL Server Service Broker.

Technologies

Although I remember the days when you had to buy 3rd party applications (like IBM MQSeries) if you wanted message queuing functionality, Microsoft has incorporated MSMQ 1.0 into Windows NT back in 1994. Fifteen years later, we are now at version 4, and MSMQ is considered to be a proven messaging platform. Even the advent of Windows Communication Foundation did not make it obsolete, since WCF includes special MSMQ binding. Service Broker is a much more recent addition to Microsoft tools lineup; it debuted with SQL Server 2005 and was updated in SQL Server 2008.

Message Size And Content Validation

MSMQ uses special memory-mapped files to store messages, while Service Broker relies on SQL Server database engine for storage. Message size limit is 4Mb and 2048Mb, respectively (Service Broker uses varbinary(max) data type to store messages). When it comes to describing the content of the message, MSMQ offers 18 "Body Type" values, such as integer, double, date, ANSI or Unicode strings, etc. but this information is optional: MSMQ doesn't validate message contents. Service Broker, on another hand, has just 3: empty, XML, and XML with schema, but it does validate the contents and rejects messages that don't conform.

Queue-Server Affinity

When developing high-performance MSMQ applications, one important best practice is "send remotely, receive locally" (here is a link to the article that explains it). What this means is that the process reading from the queue has to exist on the same machine where the queue is located. This impacts overall system scalability, because it makes adding additional processing servers more difficult.

Service Broker doesn't have such limitations, although it does require an instance of SQL Server database engine. In enterprise applications, it is generally a good idea to use a dedicated instance and host all message queues there. Applications that send and receive messages will use this instance as a global hub; they can be physically located on different servers.

Programmability

There are a number of ways to develop MSMQ applications: you can use native API, COM, or .NET. Programmers can use literally any language, from VB to C# to C++. Service Broker is less accessible: its main language is Transact-SQL and in order to use it in, say, a .NET application, you will need to write an ADO.NET wrapper. SQL Server programming samples include one such wrapper, Microsoft.SqlServer.Broker.dll, which exposes classes like SsbService and SsbMessage that shield developers from Transact-SQL.