Two Strategies For Asynchronous Message Processing

Asynchronous Message Processing

Asynchronous message processing is a common pattern in designing highly available web applications. Essentially, it means that certain tasks are not processed immediately within the scope of a HTTP request. Instead, they are handled asynchronously, by a different application unit, which in Microsoft realm is often (but not always) implemented as a Windows service. This helps streamline the web application and make it appear faster to the end user.

There is a special class of tasks that are suitable for asynchronous message processing, and it is different from typical scheduled jobs (also executed by a separate application service). Usually, they are closely related to user interactions with the main web application, but are too process-expensive or slow to be incorporated into the page processing workflow. Dynamic rendering of a multi-page PDF report is a good example since it falls under both categories. Sometimes, the tasks are not necessarily slow, but they do not produce any information for the user, for example keeping an audit trail of user activity within the system.

Regardless of the specific task being "outsourced", web application exchanges information with the processing service using discrete messages. In a typical design, messages flow only one way: from web application to the service (although there are scenarios where two-way dialog is preferable). There are two main strategies for implementing this design, MSMQ and SQL Server Service Broker.

Technologies

Although I remember the days when you had to buy 3rd party applications (like IBM MQSeries) if you wanted message queuing functionality, Microsoft has incorporated MSMQ 1.0 into Windows NT back in 1994. Fifteen years later, we are now at version 4, and MSMQ is considered to be a proven messaging platform. Even the advent of Windows Communication Foundation did not make it obsolete, since WCF includes special MSMQ binding. Service Broker is a much more recent addition to Microsoft tools lineup; it debuted with SQL Server 2005 and was updated in SQL Server 2008.

Message Size And Content Validation

MSMQ uses special memory-mapped files to store messages, while Service Broker relies on SQL Server database engine for storage. Message size limit is 4Mb and 2048Mb, respectively (Service Broker uses varbinary(max) data type to store messages). When it comes to describing the content of the message, MSMQ offers 18 "Body Type" values, such as integer, double, date, ANSI or Unicode strings, etc. but this information is optional: MSMQ doesn't validate message contents. Service Broker, on another hand, has just 3: empty, XML, and XML with schema, but it does validate the contents and rejects messages that don't conform.

Queue-Server Affinity

When developing high-performance MSMQ applications, one important best practice is "send remotely, receive locally" (here is a link to the article that explains it). What this means is that the process reading from the queue has to exist on the same machine where the queue is located. This impacts overall system scalability, because it makes adding additional processing servers more difficult.

Service Broker doesn't have such limitations, although it does require an instance of SQL Server database engine. In enterprise applications, it is generally a good idea to use a dedicated instance and host all message queues there. Applications that send and receive messages will use this instance as a global hub; they can be physically located on different servers.

Programmability

There are a number of ways to develop MSMQ applications: you can use native API, COM, or .NET. Programmers can use literally any language, from VB to C# to C++. Service Broker is less accessible: its main language is Transact-SQL and in order to use it in, say, a .NET application, you will need to write an ADO.NET wrapper. SQL Server programming samples include one such wrapper, Microsoft.SqlServer.Broker.dll, which exposes classes like SsbService and SsbMessage that shield developers from Transact-SQL.

Book Review: "The Wikipedia Revolution"

"The Wikipedia Revolution" by Andrew Lih tells the short but enchanting story of Wikipedia: how the project conceived as a fairly regular for-profit web enterprise evolved into something dramatically different. In just a few short years after Nupedia creators embraced wiki-wiki-web and opened the site to everyone with a browser, Wikipedia grew to be one of the top 10 most visited sites on the Internet. It has unparalleled reach with its 259 supported languages, and at least 25 of these languages include over 100,000 articles. All of this has been achieved with essentially all-volunteer force from around the world.

There are some aspects of the book I didn't like very much. For example, it looks like the author has applied one of the Wikipedia editing principles: "No original research". All the information in the book is compiled from publicly available sources; there are no new interviews. Numerous biographies of geeks are rather boring, and most of the additional material about Linux, Free Software Foundation, Mozilla, etc. belongs in sidebars, not main text. (I was reading Chris Anderson's "Free" at the same time and it is amazing how much of that general material is echoed in both books.)

That being said, I am positive you will learn something new from "The Wikipedia Revolution" (unless you are a seasoned wikipedian, of course). Like the intricacies of maintaining three different scripts of the same language (check out Kazakh Wikipedia), for example. Or what roles in the organization are played by administrators and bots. Or what happened when someone googled the word "jew" and didn't like what he saw.

But in my opinion, the most interesting part discusses various controversies surrounding Wikipedia. How does a quality of articles produced by countless anonymous contributors compare with the quality of established encyclopedias, such as Britannica? How does the open system protect itself from vandalism and libel without becoming a closed system? How many articles is too much? (Does every high school need to have an entry? What about elementary schools?) Last but not least, what happened when it was revealed that a prominent Wikipedia contributor and administrator falsely pretended to be a university professor?

Enjoy the book!

Exposing EntLib to COM Clients

All of us (well, most of us) know and appreciate the benefits of Enterprise Library (EntLib) Application Blocks: they solve common problems, encapsulate best practices, implement design patterns. They are easy to use and not hard to extend. What's even better, they ensure consistency across different applications and development teams.

Sadly, but all this goodness is only available to .NET code.

I do not have exact statistics, but anecdotal evidence suggests that even software companies firmly committed to .NET platform still have 25-50% of their codebase in C++, VB 6, or some form of VBScript. The ratio will continue to shift but legacy code is unlikely to disappear anytime soon (after all, mainframes and COBOL are still with us). And of course, all that code needs to be maintained.

Programmers are rarely enthusiastic about legacy code maintenance. Part of the reason is that such code is often a reverse of EntLib: it doesn't encapsulate best practices, doesn't use design patterns, is difficult to use. Yet, we cannot afford to rewrite the whole thing and have to be content just patching holes. Therefore, I think many will welcome the possibility to somehow plug in Application Blocks into their legacy code. Parts of EntLib aren't useful in the non-.NET world, of course, but things like logging, caching, and cryptography are perfect integration points. 

In the remaining part of this post, I will describe how this can be done and will use Logging Application Block as an example.

COM Facade

There are many different ways to expose EntLib functionality to older applications, but COM Interop is probably the most efficient. The idea is to use a "facade" design pattern: create a .NET class that will be exposed to COM clients via Interop and pass through calls to EntLib.

I decided to derive this new facade from System.EnterpriseServices.ServicedComponent and host it in a COM+ server application. This way we can take advantage of a couple of very useful services provided by COM+ infrastructure.

[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
public class EntLibAdapter : ServicedComponent, IEntLibAdapter

Interface IEntLibAdapter contains a single method:

[ComVisible(true)]
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
public interface IEntLibAdapter
{
void  WriteLog(ref ILogData log, string category, int priority);
}

Here, ILogData is a name of another interface that defines multiple properties collected for logging. Our COM clients will invoke WriteLog method of the facade class and Logging Application Block will determine whether to log or not (based on category and priority), and which logging trace listener to use. This, however, requires that Application Block be properly initialized.

Initialization

EntLib application blocks rely on XML configuration which is usually stored in the web.config or exe.config file. In our scenario, entry point is not a .NET application, so we cannot rely on default behavior. Fortunately, EntLib supports multiple configuration sources: for example, it can consume a stand-alone file and read XML configuration from it. Here is how this is achieved for Logging Application block:

FileConfigurationSource configSrc = new FileConfigurationSource(fileName);
LogWriterFactory factory = new LogWriterFactory(configSrc);
this.LogWriter = factory.Create();

Because the facade class is a ServicedComponent, we can take advantage of the COM+ Activation service. When object is created, COM+ will invoke Construct method and pass a string that in our case will contain full path to the configuration file:

[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
[ConstructionEnabled(true, Default = @"C:\EntLibAdapter.config")]
public class EntLibAdapter : ServicedComponent, IEntLibAdapter
{
protected override void Construct(string s)
{
 if (File.Exists(s))
 {
     FileConfigurationSource configSrc = new FileConfigurationSource(s);
     LogWriterFactory factory = new LogWriterFactory(configSrc);
     this.LogWriter = factory.Create();
 }
}
}

Object Pooling

In a typical usage scenario, EntLibAdapter object will be constructed, WriteLog method invoked, and then the object will be destroyed. Notice that the initialization step is fairly expensive - it involves reading a file from disk, parsing XML, building up objects. We can improve performance and increase overall system scalability by maintaining a pool of objects. That way EntLibAdater instances do not get destroyed after client releases the reference - they simply go back to the pool. Object pooling is another built-in service in COM+, so all we need to do is mark our class to participate:

[
ComVisible(true),
ClassInterface(ClassInterfaceType.None),
ConstructionEnabled(true, Default = @"C:\Nexsure\Installation\Dlls\EntLibAdapter.config"),
EventTrackingEnabled(true),
ObjectPooling(true, MinPoolSize = 5)
]
public class EntLibAdapter : ServicedComponent, IEntLibAdapter

Object pooling may not be the best approach if you plan to use flat file logging. In the example above, there will always be 5 instances in the pool, and each will create its own log file (4 of them will have a GUID-based file name). This should not be a problem if your primary target is database or Windows Event Log.

Deployment

Since EntLibAdapter is a ServicedComponent, it has to be strongly named and registered using RegSvcs.exe. In addition, EntLib assemblies that it depends on, such as Microsoft.Practices.EnterpriseLibrary.Logging.dll, need to be placed in the GAC.

Design For Operations, Part III - MMC Integration

Back in 2006 I started writing about designing applications with IT department in mind. I'm glad to say that my position didn't change since then - I still believe computer software should be friendly to its end users, which are either customers (mobile, desktop and web apps) or IT engineers (web apps and services). Unfortunately, these two groups are not represented equally in the design process: business analyst is a voice of the customer, but who is a voice of IT?

MMC Background

In my previous posts I covered such aspects as event logging, performance counters, and WMI integration. Today I wanted to discuss Microsoft Management Console (MMC) which was originally developed for Windows NT Option Pack. The idea was to create a common interface for managing IIS, Certificate Server, and Transaction Server, so that administrators have fewer tools to learn. This was further extended in the next release, MMC 2.0 (included with Windows XP/Server 2003), when a concept of the snap-in was added. Snap-in was a COM in-process server that MMC would communicate with, thus allowing any third-party application to have a custom management screen within MMC. In order to develop custom snap-ins, you would need to be well-versed in C++ COM development, as there were 30-something interfaces that could be used. Development of snap-ins in managed code was not supported, although there were custom frameworks, for example, an open-source project called MMC.NET.

MMC 3.0, which was shipped with Vista and Server 2008 (but available for download for older platforms), includes a managed layer and thus natively supports snap-in development in any .NET language. Everyone but hard-core C++ programmers would agree that this can be done faster, with fewer lines of code and simplified maintenance. And even they will have to admit that the ability to use WinForms inside MMC is really cool.

Setting Up Solution

In order to develop our custom MMC snap-in using C# 2008, we will create a class library project. However, before you begin, there is one important step: executing %WINDIR%\System32\MMCPerf.exe from the command prompt. This will put MMC assemblies in the GAC and NGEN them. After creating new project, add a reference to Microsoft.ManagementConsole.dll by browsing to the following folder: \Program Files\Reference Assemblies\Microsoft\mmc\v3.0\. Unfortunately, the assembly doesn't appear in the .NET tab of the "Add Reference" dialog.

Writing Code

We will start by adding two classes to the project: one derived from Microsoft.ManagementConsole.SnapInInstaller, will be used to register custom snap-in with MMC, and another, derived from Microsoft.ManagementConsole.SnapIn, will serve as an entry point.

using System.ComponentModel;
using System.Security.Permissions;
using Microsoft.ManagementConsole;

[assembly: PermissionSetAttribute(SecurityAction.RequestMinimum, Unrestricted = true)]

namespace Microsoft.ManagementConsole.Samples
{
[RunInstaller(true)]
   public class InstallUtilSupport : SnapInInstaller
{
}

[SnapInSettings("{9627F1F3-A6D2-4cf8-90A2-10F85A7A4EE7}",
DisplayName = "- Sample SnapIn",
Vendor = "My Company",
Description = "Shows FormView")]
   public class SelectionFormViewSnapIn : SnapIn
{
}
}

Note the attribute that decorates SelectionFormViewSnapIn class. All snap-ins are defined in the  Registry, so we have to provide a GUID, and also specify metadata which will be displayed in the catalog. We can leave the body of the installer empty, but SnapIn class requires a constructor.

public SelectionFormViewSnapIn()
{
    // Create the root node.
    this.RootNode = new ScopeNode();
    this.RootNode.DisplayName = "Selection (FormView) Sample";

    // Create a form view for the root node.
    FormViewDescription fvd = new FormViewDescription();
    fvd.DisplayName = "Users (FormView)";
    fvd.ViewType = typeof(FormView);
    fvd.ControlType = typeof(SelectionControl);

    // Attach the view to the root node.
    this.RootNode.ViewDescriptions.Add(fvd);
    this.RootNode.ViewDescriptions.DefaultIndex = 0;
}

In the constructor, we are effectively defining the root node of the snap-in. In this example, we will be using a WinForms user control called SelectionControl. This is a regular UserControl that implements a special interface Microsoft.ManagementConsole.IFormViewControl, which really only has a single method: void Initialize(FormView view). This is where we would put our initialization logic.

Deployment

After successful build of the solution, we will end up with a single DLL. Deploying it is very straightforward: all you need to do is execute InstallUtil.exe against it. Assuming you didn't skip the first step (running MMCPerf.exe), you should see no errors. Start MMC and choose "Add/Remove Snap-in" from the File menu, then click the "Add" button. You should see your custom snap-in appear in the list.

How To Force Partial Postback

ASP.NET UpdatePanel control allows us to program web pages with partial postback. The page still goes through most of its lifecycle stages, but only a portion of HTML is updated in the browser. This is a quick and efficient way to improve user experience by reducing page flicker.

Usually, the events that trigger partial postback of an UpdatePanel are defined declaratively in .aspx file. For example, events by child controls are considered triggers by default. If a control is located outside UpdatePanel, it can still be declared a trigger using markup:

<asp:UpdatePanel ID="MyUpdatePanel" runat="Server">
<Triggers>
  <asp:AsyncPostBackTrigger ControlID="Button1" />
</Triggers>
</asp:UpdatePanel>

However, sometimes it is necessary to trigger partial postback programmatically. For example, consider an UpdatePanel that is used together with CollapsiblePanelExtender. We want to force postback when user expands the panel. The logical place to do so is inside the "Expanded" event handler, and the code is very simple:

function onCollapsiblePanelExpanded(sender, args) {
__doPostBack('<%= MyUpdatePanel.ClientID %>', '');
}

If we need specific server-side logic to handle partial postback, ScriptManager control offers two properties: bool IsInAsyncPostBack and string AsyncPostBackSourceElementID. The former indicates whether page is processing a partial postback, and the latter contains ID of the UpdatePanel being processed.

Using Web Client Software Factory With Mobile Web Forms

After I somewhat successfully solved the problem Visual Studio 2008 has with mobile web forms, I had to tackle another challenge. The website is built with Web Client Software Factory, a powerful and flexible ASP.NET framework from Microsoft Patterns & Practices team. WCSF, or, more specifically, CWAB (composite web application block) provides dependency injection to the application. Unfortunately, the most recent release of WCSF doesn't include any support for mobile web forms. Of course, this release is almost one year old, and I know that good people of P&P are planning a new release for 2010, so hopefully this will be addressed, but in the meantime here is the solution I came up with.

WCSF guidance package includes a nice set of recipes that automate creation of web forms, master pages and user controls. The boilerplate code that is autogenerated for code-behind classes defines them as derived from Microsoft.Practices.CompositeWeb.Web.UI.Page class instead of standard System.Web.UI.Page:

public partial class MySummaryView : Microsoft.Practices.CompositeWeb.Web.UI.Page, IMySummaryView
{
}

The Page class overrides the OnPreInit method, and that is where dependency injection "magic" happens:

protected override void OnPreInit(EventArgs e)
{
 base.OnPreInit(e);
 Microsoft.Practices.CompositeWeb.WebClientApplication.BuildItemWithCurrentContext(this);
}

We can use similar approach for mobile web forms. Of course, there is no class in WCSF that is derived from System.Web.UI.MobileControls.MobilePage, so we will have to create our own base class in App_Code (or in a shared class library):

namespace Microsoft.Practices.CompositeWeb.Web.UI
{
    public class MobilePage : System.Web.UI.MobileControls.MobilePage
    {
        protected override void OnPreInit(EventArgs e)
        {
            base.OnPreInit(e);
            Microsoft.Practices.CompositeWeb.WebClientApplication.BuildItemWithCurrentContext(this);
        }
    }
}

Now, as long as we derive our mobile web form from this class, we can declare a Presenter property and let CWAB inject it at runtime. There is one drawback, though: we still need to manually create the presenter class and view interface, something that the guidance package recipe used to do automatically. Unfortunately, I don't know GAT well enough to create my own recipe for mobile web form, so the workaround I am using is this:

1. Execute "Add page with presenter" recipe
   
2. Modify .aspx file to register "mobile" tag prefix
3. Modify code-behind file to change the base class

Mobile Web Forms in Visual Studio 2008

I recently discovered that Visual Studio 2008 dropped support for ASP.NET mobile. You can create ASP.NET websites, of course, but try adding a mobile web form or mobile user control - these item templates are no longer there.

Omar Khan wrote a post for Visual Web Developer Team blog which describes a workaround but doesn't explain why this happened in the first place. One big problem with that workaround is that you can't download it due to a broken link.

I found that one way to solve the problem is to copy mobile item templates from Visual Studio 2005 (assuming you still have it installed) to a special folder where VS 2008 will look for user item templates. Here's the detailed how-to:

1) Find item templates in VS 2005 folder:

2) Find user item templates folder:

3) Copy MobileWebForm.zip, MobileWebUserControl.zip, and MobileWebConfig.zip to that folder

4) Restart VS 2008. Mobile web items now appear in the "Add New Item" dialog under "My Templates":

One problem still remains: VS 2008 designer doesn't display mobile forms and controls. This isn't a major issue for me because I hardly ever use the designer.

Cloud Storage or SDS?

Ever since Ray Ozzie has announced Windows Azure on last year's PDC (watch the keynote) there's been a lot of buzz about new platform. Every recent Microsoft event, it seems, included a session or two on Azure. SoCal Code Camp that took place this past weekend at CalState Fullerton had an entire track of cloud-related presentations. 

One general observation: details of the new platform are still, well, cloudy. Windows Azure is presently in a CTP stage; nobody expects an RTM until the end of 2009. There are a lot of technology-, process-, and cost-related questions that no one yet knows the answers to. What's worse, marketing geniuses at Microsoft decided to slap Azure label on a set of technologies that originated in different parts of the company (and even Microsoft evangelists admit that there is very little coordination).

Take a look at the obligatory Azure platform stack slide:

My initial assumption about SQL Server Data Services (SDS) was that it is somehow built on top of Azure. Apparently, SDS is a completely separate service. In fact, you don't even need to have an Azure application in order to use it.

Let's take a closer look at the data support for cloud applications. This is, in my opinion, the biggest paradigm shift for developers and architects. After all, it's easy to understand the concept of deploying your application code to a whole bunch of virtual servers, but how are we going to survive without our beloved connection strings, stored procedures, triggers?

There are two options available to us, cloud storage and SDS. Both are going to be reliable, scalable, highly available, and support terabytes of data. On the back end, both will utilize a vast  network of SQL Server nodes that use some advanced algorithms to support distributed data storage and replication. Below is a side-by-side comparison.

Signup. When you sign up for Azure, you receive a separate storage account. To use SDS you will need to get yet another account.

Hierarchy

Cloud Storage: Provides account/container/entity model for your data.

SDS: The model is similar - authority/container/entity

Data Abstractions

Cloud Storage: Supports blobs (basically, named files with metadata) up to 50Gb, tables (which are essentially lists of entities, not database tables), and queues with message size up to 8Kb.

SDS: Only works with entities, which are similar to the tables above.

Data Access

Cloud Storage: Blobs and queues can be accessed via REST, but tables are also exposed via ADO.NET Data Services. This allows for a more convenient API (for example, you can query a table using LINQ). Large blobs can be uploaded by small 4Mb-sized chunks.

SDS: Entities can be queried using REST. Although there is no ADO.NET Data Services support, you can pass a LINQ-style query in the HTTP request (query may even join entities).

*** UPDATE (March 17, 2009)

Someone at Microsoft have finally noticed the striking similarities between Cloud Storage and SDS. Data Platform Insider blog is announcing the change to SDS architecture: REST-based interface will be decommissioned and replaced by a service protocol based on Tabular Data Stream (which has been a SQL Server network protocol since SQL 2000). Public CTP of the new architecture will be available in the middle of this year.

Analytical Approach To Solving Programming Problems

In the six months that passed since I updated this blog I've been working on various web application projects, learning a lot about ASP.Net Ajax and Web Client Software Factory. Nevertheless, this posting isn't about any particular technology. In my opinion, software developers already have way too many technologies, frameworks, programming languages, and APIs available to us. It's a challenge just to keep up with all the new stuff that comes out. What I want to discuss instead are the benefits of the analytical approach to programming problems.

Here is a sample problem. Imagine there is a virus spreading through the cells of a very large two-dimensional matrix. We start with a relatively healthy matrix with only 10 random cells infected. The virus is spreading by infecting 4 adjacent cells every minute. For example, if "." represents a healthy cell, this is how the epidemic will progress:

Start ......... ......... ....0.... ......... .........

After first minute ......... ....1.... ...101... ....1.... .........

After second minute ......... ....2.... ...212... ..21012.. ...212... ....2.... .........

Of course, the virus starts spreading from 10 different places on the surface, so depending on where these cells are, the time it takes to infect entire matrix can vary. Our task is to find that time given 10 initial locations.

It may be tempting to rely on a raw processing power of modern computers and concoct a solution that looks like this:

while (!matrix_is_fully_infected)
{
infect_next_set_of_cells();
}

The model above simply recreates the behavior of the virus. The obvious drawback here is the sheer inefficiency of the algorithm: we end up scanning entire matrix an unknown number of times. As matrix size increase, the inefficiency will be more evident. Still, this may be a valid approach in some cases, where there is no easy analytical solution. Fortunately, our virus has a primitive DNA and yields itself to mathematical definition.

For simplicity, let's assume that we begin with a single infected cell with coordinates (a,b). The number of minutes it takes to infect an arbitrary cell (x,y) can be expressed with this simple formula: |a-x|+|b-y|. Now let's assume we had a second infected cell at the beginning: (c,d). We could use a similar formula to find out how many minutes it will need to infect our arbitrary cell (x,y): |c-x|+|d-y|.

Depending on whether (a,b) or (c,d) is located closer to (x,y), one of the above expressions will produce a smaller number of minutes. This will be the answer to the question "how long it takes to infect a single arbitrary cell". As we go from 2 infected cells to the original 10, we can write the answer as a function of (x,y):

min(|a_i - x| + |b_i - y|), where 1 <= i <= 10

Of course, our job is not done yet - the virus doesn't stop until all cells are infected. What we need to find out is how many minutes it will take to infect the last cell. Evidently, this will be the maximum time across the matrix, so our solution will be to take the maximum of the above function:

max( min(|a_i - x| + |b_i - y|) ), where x and y vary across matrix dimensions

As you can easily see, analytical approach provided significant performance improvement - we now only need to scan the matrix once.

Automatic Deployment of SQL Scripts

1. Background

In my article "Introduction To Change Management" I wrote about the fundamental flaw in the deployment process:

Developers have the first-hand knowledge about the deployment artifacts, but they rarely have security privileges on servers and databases outside of the development environment. DBA's and operations, on the other hand, have permissions and are usually in charge of performing deployments, but they rarely have a good understanding of the deployment artifacts and their relationships.

Suppose we need to deploy a new build of assembly Foo.dll. A method inside it relies on a specific version of stored procedure dbo.Bar, so updated SQL script must be deployed together with the assembly. This is where our process becomes vulnerable to human errors, since IT, DBA, build engineer, and release coordinator all have an opportunity to break the deployment. Take into consideration the sheer number of components and database objects, and you can understand why it usually takes several iterations to make a successful push.

The standard two-track deployment process has other drawbacks, too. For example, it turns DBAs into very expensive clerks as they mindlessly combine scripts together and press "F5" button. Another problem is that it creates a false illusion that database scripts can be safely deployed as a hotfix (while binary code requires regression testing). In reality, both application and database code must be treated as a single entity.


2. Solution

My solution is to bundle all SQL scripts that are required by an assembly with the assembly itself using embedded resources. During deployment, IT engineer will need to invoke standard .NET installation utility:

Installutil.exe <assemblyname>.dll

This invokes a custom installer class which will enumerate script resources in the assembly and execute them in a predefined order.

Although it sounds pretty simple, the above approach has a potential to eliminate a lot of deployment issues, because it gives the developer - someone intimately familiar with implementation details - full control over database deployment process.

It also encourages a modular approach to software design by reducing external dependencies of the assembly.


3. How To Bundle Scripts With Assembly

3.1 Adding resources.

Adding SQL scripts to assembly is a very straightforward procedure. In Visual Studio Solution Explorer, right-click project name and choose "Add Existing Item" from the context menu (it is even better to create a subfolder for these files). After a file was added, open its properties and change Build Action from "None" to "Embedded Resource".

3.2 Marking resources as SQL scripts.

When deploying SQL scripts, order is very important. For example, if you have a stored procedure which selects records from table dbo.Customers, you have to create the table before you can create the stored procedure. So, in order to maintain the order, I created a new custom attribute SqlScriptResourceAttribute:

[AttributeUsage(AttributeTargets.Assembly, Inherited = false, AllowMultiple = true)]
public sealed class SqlScriptResourceAttribute : Attribute
{
   public SqlScriptResourceAttribute(string scriptName, int scriptSequence)
   {
      this.ScriptName = scriptName;
      this.ScriptSequence = scriptSequence;
   }
}

When using the attribute, it is important to specify fully qualified name of the script, which is assembly name + subfolder name (if any) + file name. In the example below, table creation script t_Customer.sql will be executed before other scripts.

[assembly: SqlScriptResource("MyAssembly.pr_Customer_s.sql", 2)]
[assembly: SqlScriptResource("MyAssembly.pr_Customer_i.sql", 3)]
[assembly: SqlScriptResource("MyAssembly.t_Customer.sql", 1)]

3.3 Adding Custom Installer

In order to be able to run installutil.exe against your assembly, you need to add a custom installer to it. This is done using a few lines of code:

using System.ComponentModel;
using System.Configuration.Install;

namespace MyAssembly
{
   [RunInstaller(true)]
   public class MyAssemblyInstaller : Installer
   {
      public MyAssemblyInstaller()
      {
         ScriptInstaller installer = new ScriptInstaller(this.GetType().Assembly);
         this.Installers.Add(installer);
      }
   }
}

All we need to do is to initialize an instance of ScriptInstaller class with the reference to current assembly and add it to the Installers collection.


4. How To Implement ScriptInstaller

ScriptInstaller is a subclass of System.Configuration.Install.Installer. It overrides standard Install(IDictionary stateSaver) method and implements SQL script deployment logic.

4.1 Enumerating Scripts.

I use Reflection to retrieve embedded resource names and list of SqlScriptResourceAttribute instances, then cross-check them to ensure integrity. Finally, I add all script resource names to a SortedList, ordering them by execution sequence.

private SortedList<int, string> BuildScriptList()
{
   SortedList<int, string> scripts = new SortedList<int, string>();
   List<string> resources = new List<string>(_currentAssembly.GetManifestResourceNames());

   object[] attributes = _currentAssembly.GetCustomAttributes(typeof(SqlScriptResourceAttribute), false);
   if (attributes != null)
   {
      foreach (object item in attributes)
      {
         SqlScriptResourceAttribute attrib = (SqlScriptResourceAttribute)item;
         if (resources.Contains(attrib.ScriptName))
         {
            scripts.Add(attrib.ScriptSequence, attrib.ScriptName);
         }
         else
         {
            Context.LogMessage(string.Format("## Script {0} not found in the current assembly.", attrib.ScriptName));
         }
      }
   }
   return scripts;
}

4.2 Loading Scripts.

Loading scripts from the assembly is accomplished using a GetManifestResourceStream method of the Assembly class:

foreach (string scriptName in BuildScriptList().Values)
{
   Context.LogMessage(string.Format("## Installing script: {0}", scriptName));
   Stream resourceStream = _currentAssembly.GetManifestResourceStream(scriptName);
   string sqlScript;
   using (StreamReader sr = new StreamReader(resourceStream))
   {
      sqlScript = sr.ReadToEnd();
   }
   DeployScript(sqlScript);
}

4.3 Deploying Scripts.

Although this is essentially a standard ADO.NET ExecuteNonQuery() operation, there are two potential caveats. First is database security. Whichever way your code normally builds database connection string, it is very unlikely you will be able to use it with DDL scripts. Commonly, you only get "execute" (and possibly "select") permissions at runtime. So, in order to successfully execute deployment scripts, you need to tweak the connection string to establish a more privileged security context. The best approach, in my opinion, is to use SQL Server Integrated Security and assume that the person executing the installutil command has relevant SQL Server permissions.

Second potential pitfall is related to the fact that most deployment scripts contain multiple batches of TSQL separated by the "GO" command, e.g.:

IF  EXISTS (SELECT * FROM dbo.sysobjects WHERE id = OBJECT_ID(N'[dbo].pr_Customer_s') AND OBJECTPROPERTY(id,N'IsProcedure') = 1)
DROP PROCEDURE [dbo].pr_Customer_s
GO

CREATE PROCEDURE dbo.pr_Customer_s

Keyword "GO" is not part of TSQL language definition, so if you try to call ExecuteNonQuery on the above script directly, you will get an exception. What you need to do is split the script into individual batches using "GO" keyword as a guide, and then execute each batch individually:

string[] batches = sqlScript.Split(new string[] { "\r\nGO\r\n" }, StringSplitOptions.RemoveEmptyEntries);
foreach (string sqlBatch in batches)
{
   DbCommand cmd = db.GetSqlStringCommand(sqlBatch);
   db.ExecuteNonQuery(cmd);
}

5. Wrapping Up

There are situations where the solution described in this article is either not applicable (for instance, when company ships their software on a CD with a special setup application) or doesn't add value (for example, when database engineers - not .NET programmers - are developing database scripts). But if your change management process separates database scripts from binary code deployment, you may have an opportunity to streamline it. Automated scripts installation will reduce the number of deployment issues, and DBAs will surely feel relieved that they no longer need to perform this mundane task.

WCSF 2.0 And Multiple Websites

Patterns and Practices team just released version 2 of the Web Client Software Factory. Among new features are implementation of MVP pattern for user controls and master pages, and a few AJAX extender controls.

PageFlow is not present in this release, and its future seems to be murky. I spent some time trying to implement PageFlow in my project and was disappointed by its complexity. Benefit to effort ratio was definitely too low; I eventually gave up on it.

I have followed the steps outlined in documentation in order to migrate my solution from June 2007 (1.1) release of the factory without much trouble. Unfortunately, there is still no recipe to add new website to the solution. So, if you need to have more than one, make a copy of the existing website and add it. Make sure little file called vwd.webinfo is present in the root of the website, otherwise WCSF will not "recognize" it - new site name will not appear in drop-down lists or will be grayed out when you execute various recipes from the guidance package. I don't believe this tip is documented.

LINQ To SQL Performance

In the interest of full disclosure, I must say that I like LINQ. I think the declarative approach to data manipulation is just wonderful! It makes code much cleaner and code maintenance much easier. There is an overhead, of course, but with LINQ to Objects all calls are in-process, so it's still going to be fast.

When it comes to LINQ to SQL, though, the game changes somewhat: not only database access is out-of-process, it is also notoriously tricky. Poorly designed queries can take minutes instead of seconds and drain server resources, so it is very important to know what T-SQL is being generated for a given LINQ query. But that is a topic for another post...

What I really wanted to know is - all other things being equal - how much performance overhead does LINQ to SQL add on top of ADO.NET. So, I built my test harness around one simple query against Northwind database:

SELECT [t0].[OrderID], [t0].[OrderDate], [t0].[CustomerID], [t1].[CompanyName], (
 SELECT COUNT(*)
 FROM [dbo].[Order Details] AS [t2]
 WHERE [t0].[OrderID] = [t2].[OrderID]
 ) AS [ProdCount]
FROM [dbo].[Orders] AS [t0]
INNER JOIN [dbo].[Customers] AS [t1] ON [t0].[CustomerID] = [t1].[CustomerID]
WHERE [t0].[OrderID] = @OrderID

Basically, for a given order ID, we are retrieving a single row with information from order, customer, and order details tables:

OrderID     OrderDate               CustomerID CompanyName                              ProdCount
----------- ----------------------- ---------- ---------------------------------------- -----------
10248       1996-07-04 00:00:00.000 VINET      Vins et alcools Chevalier                3

(1 row(s) affected)

In order to get measurable results, I wanted to run this query for every one of 830 orders (independently, to simulate a multi-user application). For benchmarking, I used three alternative approaches: dynamic SQL, dynamic SQL with parameters, and stored procedure. Because SQL Server caches query execution plans, I restarted it before switching to a different approach.

Dynamic SQL

In the method below, I am concatenating order ID value to the end of the WHERE clause. This technique is known to have poor performance, because SQL Server doesn't realize we are using the same query and will have to compile it every time. Indeed, the initial run took approximately 4100 ms on my laptop. SQL Server caches query plans, though, so subsequent executions of the test yielded a much better result of roughly 200 ms.

private TimeSpan RunADOTest()
{
  using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["Northwind"].ConnectionString))
  {
     conn.Open();
     SqlCommand cmd = conn.CreateCommand();
     cmd.CommandType = CommandType.Text;
     DateTime dtStart = DateTime.Now;
     for (int orderId = 10248; orderId < 11078; orderId++)
     {
        cmd.CommandText = @"SELECT [t0].[OrderID], [t0].[OrderDate], ...
          WHERE [t0].[OrderID] =  " + orderId.ToString();
        SqlDataReader dr = cmd.ExecuteReader();
        dr.Close();
   }
   return DateTime.Now.Subtract(dtStart);
  }
}

Parameterized Dynamic SQL

The only difference from previous method was that query WHERE clause changed to "WHERE [t0].[OrderID] = @OrderID". I also added these lines before calling cmd.ExecuteReader():

   cmd.Parameters.Clear();
   cmd.Parameters.Add(new SqlParameter("@OrderID", orderId));

As expected, performance has improved. After SQL Server restart, the method completed in 460 ms, and subsequent executions were around 190 ms. It's also important to realize that SQL Server has cached only one query plan and not 830 as in the previous example.

Stored Procedure

Ever since Microsoft added query plan caching for dynamic queries in SQL Server 2000, there really is no performance difference between stored procedure and a parameterized dynamic query. Of course, there are many good reasons for writing stored procedures (greater security, better code reuse, smaller network traffic).

In my test harness, results were nearly identical to parameterized dynamic SQL: initial run (after SQL Server restart) took 453 ms, and subsequent executions took 187 ms.

LINQ To SQL

I dropped Customers, Orders, and Order Details tables to the surface of Visual Studio object relational designer to create a LINQ to SQL classes. In order to ensure that all calls are executed using a single database connection, NorthwindDataContext is initialized with an open SqlConnection object. I also wanted to make sure T-SQL generated by LINQ is the same as in previous tests, so I installed SqlServerQueryVisualizer component and ran a SQLProfiler trace. Indeed, both Parameterized DSQL and LINQ to SQL tests issued the same exact exec sp_executesql command. I had to use query variable in a foreach statement to make it run (because of deferred execution).

Test results were, frankly, disappointing. After SQL Server restarted, the method took 4250 ms. Subsequent executions yielded between 3300 and 3400 ms, or more than 10 times slower than all other tests.

private TimeSpan RunLINQTest()
{
  DateTime dtStart = DateTime.Now;
  using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["Northwind"].ConnectionString))
  {
     conn.Open();
     NorthwindDataContext db = new NorthwindDataContext(conn);

     for (int orderId = 10248; orderId < 11078; orderId++)
     {
        var query = from o in db.Orders
          join c in db.Customers on o.CustomerID equals c.CustomerID
          where o.OrderID == orderId
          select new
          {
            o.OrderID,
            o.OrderDate,
            o.CustomerID,
            c.CompanyName,
            ProdCount = o.Order_Details.Count
          };

        foreach (var item in query)
        {
          string s = item.CompanyName;
        }
     }

     return DateTime.Now.Subtract(dtStart);
  }
}

Test Summary

The table below summarizes test results. As you can see, after initial query plans are cached, dynamic SQL and stored procedure have essentially the same performance. Either approach is 94% faster than LINQ to SQL.

Can't say I'm happy with these results (after all, I do like LINQ), but at this point I don't see what else can be contributing to the delay. Unfortunately, I only have Professional version of Visual Studio which doesn't include a profiler, so I can't research further. However, I welcome any comments or corrections. Source code of the test harness is posted here: http://members.cox.net/rmamedov/blog/LINQ2SQLPerformanceTest.zip.

                   Initial  Subsequent

Dynamic SQL             4100         200

Parameterized
Dynamic SQL              460         190

Stored Procedure         453         187

LINQ to SQL             4250        3350

"Can We Export This To Excel?"

Not sure how widespread this problem is, but I often get requirements from business users to add "export to Excel" capability to this or that screen. It seems no matter how much effort we put into our web applications, adding fancy server controls, AJAX, and so on, end users just don't feel comfortable until they can load the data to Excel and start massaging it.

Moving to the practical side of the issue, what is the best way to generate an Excel spreadsheet on the web server? There are several different scenarios. If you are using Microsoft.Reporting.WebForms.ReportViewer control, there is nothing else to do because the control itself supports export to Excel. Of course, the control merely renders a report, which needs to be defined and generated first.

If Microsoft Office is installed, you could add a reference to Excel COM server and have the entire object model at your disposal. This is very powerful technology, but it requires Excel to be deployed to each web server which isn't really practical. Another downside - all these out-of-process calls to COM objects are expensive in terms of system performance.

What if we want a lean, lightweight solution that doesn't rely on any additional components? Well, we could generate a text document in comma-separated format. Excel will be able to open such a document, but it will look very primitive.

Now the good news: there is a way to generate professionally looking Excel spreadsheets from a pure ASP.NET web application. What we generate is technically not a spreadsheet, but an XML stream which conforms to the Office Excel XML schema (several schemas, to be precise). This format has been supported in Excel since Office XP. It is different from another XML-based format Microsoft adopted in Office 2007, Office Open XML. The latter is represented by a set of files placed in a ZIP archive, while the former is a single uncompressed XML file.

In order to avoid manipulating strings and StringBuilders and use an object model instead, I will create a set of classes to represent different elements of the schema (Workbook, Worksheet, Cell, Row, etc.). One way of doing this would be to generate code from schema using XSD.EXE /classes syntax. Here is how WorkBook class may look like (with fields instead of properties for brevity):

public class WorkBook : IGenerateOutput
{
    public List<WorkSheet> WorkSheets = new List<WorkSheet>();
    public List<Style> Styles = new List<Style>();
    public string Author;
    public string Company;
    public string Title;

    public WorkSheet CreateWorkSheet()
    {
        WorkSheet ws = new WorkSheet(String.Format("Sheet{0}", this.WorkSheets.Count + 1));
        this.WorkSheets.Add(ws);
        return ws;
    }

    public Style CreateStyle(string styleId)
    {
        Style s = new Style(styleId);
        this.Styles.Add(s);
        return s;
    }

    public string GetOfficeXml()
    {
        // Instantiate XmlTextWriter and call GenerateOutput()
    }

    public void GenerateOutput(XmlTextWriter writer)
    {
        writer.WriteStartElement("Workbook");
        writer.WriteAttributeString("xmlns", "urn:schemas-microsoft-com:office:spreadsheet");
        writer.WriteAttributeString("xmlns:o", "urn:schemas-microsoft-com:office:office");
        writer.WriteAttributeString("xmlns:x", "urn:schemas-microsoft-com:office:excel");
        writer.WriteAttributeString("xmlns:ss", "urn:schemas-microsoft-com:office:spreadsheet");
        writer.WriteAttributeString("xmlns:html", "http://www.w3.org/TR/REC-html40");

        if (this.Styles.Count > 0)
        {
            writer.WriteStartElement("Styles");
            foreach (Style style in this.Styles)
            {
                style.GenerateOutput(writer);
            }
            writer.WriteEndElement();
        }

        foreach (WorkSheet sheet in this.WorkSheets)
        {
            sheet.GenerateOutput(writer);
        }
    }
}

WorkBook features a couple of factory methods to create new styles and worksheets; it uses chain of responsibility pattern to delegate responsibility for rendering Xml to its child elements. Other classes look very similar:

public class WorkSheet : IGenerateOutput
{
    public List<Row> Rows = new List<Row>();
    public List<Column> Columns = new List<Column>();
    public string _Name;

    public WorkSheet(string name)
    {
        this.Name = name;
    }

    public Row CreateRow()
    {
        Row r = new Row();
        this.Rows.Add(r);
        return r;
    }

    public Column CreateColumn()
    {
        Column c = new Column();
        this.Columns.Add(c);
        return c;
    }

    public void GenerateOutput(XmlTextWriter writer)
    {
        writer.WriteStartElement("Worksheet");
        writer.WriteAttributeString("ss:Name", this.Name);

        writer.WriteStartElement("Table");

        if (this.Columns.Count > 0)
        {
            foreach (Column c in this.Columns)
            {
                c.GenerateOutput(writer);
            }
        }

        foreach (Row r in this.Rows)
        {
            r.GenerateOutput(writer);
        }

        writer.WriteEndElement();
        writer.WriteEndElement();
    }
}

So, in order to export some data to Excel, all we need to do is instantiate WorkBook class, create one or more worksheets, add rows with cells, and then call GetOfficeXml() method:

        WorkBook wb = new WorkBook();
        wb.Author = "Riyad Mammadov";
        wb.Title = "Test";

        WorkSheet ws = wb.CreateWorkSheet();
        Row r = ws.CreateRow();
        r.CreateCell(CellDataType.Number, "12345");
        r.CreateCell(CellDataType.String, "Hello, World");

        string xml = wb.GetOfficeXml();

Workflow Foundation Usage Patterns. Part II.

Before I continue describing WF usage patterns, small personal announcement: I'm changing this blog's title from "Thoughts on Architecture" to simply "Technical Blog". This seems to be more prudent, since I am not always discussing software architecture and am not even an architect anymore (my new job title is development manager, but enough about me).

Workflow In Business Layer

When we follow the principles of layered architecture, we usually end up with two types of objects in the business layer: entities and workflows. Entities, of course, map more or less directly to domain entities - products, orders, claims, or what have you. Workflows, on the other hand, manipulate entities in order to implement specific business process. For example, business process of registering new customer may include creation of Client, Address, and CreditCard entities, validation of the last two, transmission of a welcome email, etc. Of course, there are many ways to skin the cat, but typically you would employ some sort of business workflow object.

What attracted me to implement business workflows using WF was the ability to compose workflows from custom activities. We often had a situation where the same process had to be customized for different partners. In the example above, a customer may belong to a partner which does not require credit card validation. Engineers used to create new version of business workflow by copy-pasting relevant method in the class and tweaking it. My idea was to create a set of basic WF activities that represent pieces of the process, then combine them together using WF designer rather than C#.

Contrary to the pattern I described in the previous post, this one doesn't represent a long-running process and workflow instances are never persisted to database. Another side benefit is simplified maintenace due to improved readability: it's much easier to understand the logic by looking at workflow designer than raw code. Workflows become, in a sense, a live technical documentation of the system, on the par with unit tests and Visual Studio class diagrams.

There are a couple of caveats with this approach. First of all, beware of performance penalty from using WF and be sure to stress-test your system. In my case, business workflow executed as part of a batch process, not web application, so I didn't have to worry about user experience. Second, keep in mind that WF world is very asynchronous, so you can't really treat the workflow like just another procedure call. You create workflow instance, assign input parameters and start it. If you want to get the results back from that instance, you need to do extra work with the runtime.

PatternShare Is Gone

I used to keep a link to PatternShare community website on the sidebar of this blog. It turns out, the site (which was maintained by PnP team) is no longer active. The reason I am blogging about this is because we spent a full hour on PnP Summit today discussing the fate of PatternShare. The key question was where PnP should be spending its efforts: on embedding relevant patterns in its tools (application blocks, factories, guidances) or educating people by writing books and creating new version of PatternShare? There were plenty of arguments on both sides. I think the consensus opinion is that education is important, but even if PnP provides the books, there is no guarantee that people will educate themselves. Programmers are more likely to use free tools. But does good tool make bad coders smarter? Or does it make them more dangerous?

I still think design patterns should be taught in colleges...

Workflow Foundation Usage Patterns. Part I.

Today during Patterns and Practices Summit Ted Neward gave a presentation on Windows Workflow Foundation. One of the points he made is that although "Workflow" as a programming concept is fairly old (arguably older than object-oriented programming), WF as a product has been out for less than a year. Thus, it is premature to talk about best practices and proven patterns.

I agree with Ted, and in this post I'd like to share my experience designing real-life systems that take advantange of WF. As an architect, I had to choose the niche for this shiny new technology. It wasn't an easy task, because WF represents extremely flexible and universal concept. On the other hand, it is a new technology, so I wanted to minimize the risk and not use WF as a cornerstone of the system. Instead, it should be pluggable, something that's easy to turn off.

Job Orchestration

Imagine an application that prepares and sends a file to several business partners, then waits for response files and processes them. Even assuming fully automated file upload/download, the whole thing can take hours or days. My original approach was to write separate jobs that handle individual tasks, such as preparing request file, processing response file, upload, and download. It worked, but it was difficult to reconstruct the business process from individual job results. In other words, we know that a job has executed successfully, but business users wanted to know where do we stand in the larger process.

In order to solve this problem, I used WF's ability to implement long-running workflows. I represent a process with a workflow that contains the sequence of job launch activities. After a job is launched, workflow idles and is persisted to SQL data storage using SqlWorkflowPersistenceService. When job completes execution, it sends a signal to workflow runtime, which loads correct WorkflowInstance from SQL and resumes it. Note that workflow itself doesn't contain any code whatsoever and can be written in pure XAML - it represents a logical sequence of jobs.

Business users were particularly pleased to see the Visio-style diagram of the workflow which clearly showed what's completed, what's running, and what's pending. For this I used the ability of WorkflowView control to save itself as an image.

[To be continued]

Mapping Framework

Data mapping is one of the key components of enterprise application integration. Whether we are in the realm of business partner integration built on top of secure file exchange, or integrating applications built on a common SOA platform, data mapping is always there.

From an application's point of view, mapping can be either inbound or outbound. Inbound mapping converts raw data (positional or delimited flat file or XML document) into objects that are native to the application. Naturally, outbound mapping represents the reverse operation: transformation of native objects (or losely typed datasets) into flat or XML data.

Although simple logic can be hard-coded inside application, this approach doesn't scale well. It's a good idea to have a framework that will allow new mapping logic to be put in place with little or no custom coding.

In order to define mapping, we need to specify mapping rules (which source data elements map to a destination element?) and, optionally, transformation (what needs to be done with source data elements in order to arrive to the destination element?). Here is how a single inbound mapping piece can be represented (fields are shown instead of properties for brevity):

public class InboundMappingPiece
{
 public string PropertyName;
 public MethodInfo MethodInfo;
 public int StartIndex;
 public int Length;
 public string XPath;
}

Let's review the fields. PropertyName designates the "destination": which property of a native object we are populating with this mapping piece. MethodInfo is a function pointer that implements transformation logic. Now the only thing missing is the source element. In order to map from positional flat file, we need to know StartIndex and Length, while XML data is easily extracted using XPath queries.

public class OutboundMappingPiece
{
 public string[] SourceElements;
   public MethodInfo MethodInfo;
   public string DestinationElement;
   public int DestinationWidth;
   public char PadCharacter;
}

OutboundMappingPiece is built in a similar fashion. We've got an array of SourceElements (in case we wanted to use many-to-one mapping), and a MethodInfo pointer for transformation logic. If our destination is XML, we need to know the name of DestinationElement, otherwise DestinationWidth and PadCharacter allow us to generate flat file output.

By matching lists of mapping pieces with type names, we can declare the map as a whole. This is how a Mapper class may look like:

public class Mapper
{
private Dictionary<String, List<InboundMappingPiece>> _InboundMap;
private Dictionary<String, List<OutboundMappingPiece>> _OutboundMap;

   // Outbound
   public string Transform(object obj) {...}
   public string TransformToXml(object obj) {...}

 // Inbound
   public T GetObject<T>(string rawData) where T : new() {...}
   public T GetObject<T>(XmlNode node) where T : new() {...}
}

Implementation of inbound and outbound transformation methods can be boiled down to iterating through lists of mapping pieces and applying them to the source data. Of course, the devil is in details, and there are lots of details to be considered: how to handle arrays, nullable types, nested types, and so on. Another interesting question is where to store mapping configuration, but I will leave it until the next post.

Book Review: "Dreaming in Code"

I took my daughter to the library a couple of weeks ago. As we were walking toward the checkout line, I realized that I needed something to read, too, and looked at the books on the nearby "New Arrivals" stand. A title "Dreaming in Code" caught my attention. Brief look at the back cover confirmed that the book was indeed about software development. A non-technical book on software development written by a journalist? You don't see that every day. I borrowed "Dreaming in Code" from the library and I'm certainly glad I did.

After spending over a decade in the industry I knew from experience that most software projects are delivered either late, way over budget, or with significantly reduced features. I have read "The Mythical Man-Month" and understood that there are dark forces at play. Still, deep in my heart I believed that somehow somewhere exists a group of people that knows exactly how to avoid all common project pitfalls. Wouldn't it be great to learn who they are and how they do it?

Scott Rosenberg's book follows the life of one project launched in the heart of Silicon Valley by none other than Mitchell Kapor, creator of Lotus 1-2-3. The idea was to create a revolutionary personal information manager that would also be cross-platform and open-source. Kapor personally financed the venture, so there was no pressure from the "suits". Some of the brightest programmers started to work for Open Source Applications Foundation. And yet the project (code-named "Chandler") had its share of disappointments, delays, and trade-offs. Six years from launch, it is currently at version 0.7 alpha 4 which contains only the calendar (original vision also includes email, tasks, notes, and contact management).

"Dreaming in Code" is much more than a chronicle of Chandler and OSAF, though. It weaves into its storyline short essays that introduce reader to concepts like open-source development, structural and object-oriented programming, methodologies such as capability maturity model and agile. The book contains quotes from Engelbart, Raymond, Knuth, Brooks, Dijkstra, and many many other outstanding people. I guarantee you will learn something by reading it.

Book's website http://www.dreamingincode.com/ has links to Amazon and Barnes & Noble.

Integrating Services on User's Desktop

.NET architects, including myself, enjoy talking about services, service-orientation, and enterprise service bus. Our emphasis is clearly on the server side. Although that is indeed very important and extremely interesting, we tend to forget that people (whom we contemptuously call "users") are mostly interested in client-side systems. These systems are predominantly intranet-based, and their sole purpose - contrary to that of their client-server predecessors - is to invoke services. Services implement core business logic for applications that range from human resources to inventory management and from accounting to business operations control. So, when we implement SOA in our enterprises, what options do we give to users?

At the very high level, there are two principal choices for user interface: web application and windows forms (winforms) applications - thin and thick clients, respectively. Until recently, web applications were the clear winners when it came to integrating services on a desktop. Obvious advantages such as small footprint and ease of deployment gave them an edge over winforms.

Then in December 2005 Microsoft Patterns & Practices group has released a Composite User Interface application block followed by Smart Client Software Factory in June 2006. Composite UI application block is a framework for building complex, event-driven, and modular winforms applications efficiently. Smart Client software factory builds on top of it, adding proven design patterns, Visual Studio automation and extensive reference implementation. These releases, combined with ClickOnce deployment subsystem in .NET 2.0, have dramatically changed the landscape for desktop services integration. I am going to compare Smart Client architecture with web applications from that angle.

Modularity.
One of the key benefits of Smart Client applications is their inherent modularity. Classic winforms programs consist of numerous forms and ASP.NET consists of numerous web pages. In Smart Client, there is a single shell form and multiple modules. UI elements are part of the module; they are created using Model-View-Presenter design pattern. From the service integration perspective, it is beneficial to use plug-in modules for different service groups.

Data Exchange.
Lack of attention to client systems usually results in proliferation of single-use applications. Users need to constantly switch from one web application to another, or from web to winform and back. Not only this is confusing, but it may also be counterproductive, because there is typically no automated way to exchange data between systems and users have to resort to copy-paste technique. Smart Client applications have built-in capability to exchange data and events between objects.

UI Consistency.
Multitude of client applications (no matter web or winform) which are produced by different development teams makes it very difficult to maintain UI consistency. On the other hand, modular approach of Smart Client allows engineering teams to work independently on a single solution, thus reusing code and applying consistent UI.

UI Quality.
Having spent many years developing web applications, I am convinced that the quality of their user interface will never measure up to that of windows applications. Sure, ASP.NET 2.0 has many improvements, but it is the underlying platform that has problems. HTML was not designed for UI rendering, although this has been somewhat improved with CSS. Add a requirement to support different web browsers (with their different user settings), and we end up with a lowest common denominator. Naturally, there are third party UI libraries for ASP.NET but they are a) expensive and b) usually available for winforms as well.

Web Server Considerations.
SOA must be designed and deployed with optimal balance between scalability and performance. When we develop client-side system as a web application, we are essentially adding another service to our environment. Yet, its design is rarely as rigorous as the design of main services. As a result, users of the application may experience problems caused by poor server-side design. Smart Client, by contrast, doesn't rely on any other services except main SOA.

Security.
Web applications built using ASP.NET 2.0 have at their disposal such security features as membership and role providers, security controls. These are typically linked to a custom database schema. Smart Client applications have built-in authorization mechanisms: for example, it is possible to limit access to a module, command, or view by user role. No dedicated database is needed since we can get membership information for the current security principal from the organization's Active Directory. By combining multiple modules into a unified application, we are in effect providing a single-sign-on functionality.

Deployment.
Anyone who had deployed .MSI packages or setup CDs to even a small number of users will tell you that ease of deployment is the biggest benefit of web applications. Indeed, once you post updated code to the web server, there is nothing more to do: next time user accesses the web site, he or she will execute the latest version of the application. ClickOnce technology, which has debuted in 2.0 version of .NET framework, brings similar experience to the world of winforms systems. ClickOnce has several deployment patterns: for example, it allows applications to be published to a file share or URL. If we choose the online-only mode, users will have to launch the program from that location (this is very similar to web applications). However, we may allow application to be available offline. This way, the binaries will be downloaded to user's computer and program title will appear in Start menu and Control Panel. Every time the application is launched, it will check for updates and prompt user to download latest code if available.

Hopefully, I managed to convince you that Smart Client applications are a viable alternative to web when it comes to integrating services on user's desktop. The only way to find out if the technology is a match for your specific requirements, is to give it a try.

Good luck!

Introduction to Change Management

My article "Introduction to Change Management in an Application Service Provider Environment" has been published on ASPToday about a year ago. They promised to pay for it but never did, so I decided to make the article available to general public (you can still access it on ASPToday.com if you have a subscription). Enjoy: http://members.cox.net/rmamedov/ChangeManagement.htm.