Friday, November 13, 2009

Why I Hate Entity Framework

A few months ago I took over development of an insurance-related system. It was a Windows Forms application created for a small workgroup. Original design utilized ADO.NET Entity Framework and it was my first serious encounter with the technology. Immediately, I sensed that something wasn't right: application was too slow. There were only a dozen or so tables and hardly any data in them, yet forms took whole seconds to load. Database updates were even more problematic - not only they took long time, but sometimes they failed for no apparent reason.

Initially, I tried to write it all off as part of a learning curve. I recreated entity model and ran a SQL Profiler in an effort to better understand the technology. Profiler results were simply shocking: instead of executing a half-dozen or so SELECT statements in response to user opening a record, Entity Framework generated hundreds of them (trace file was over 1Mb in size).

So, I remembered the little performance test I did for LINQ to SQL about two years ago and decided to expand it to include Entity Framework. New method looks similar to the one used to test LINQ to SQL:
private TimeSpan RunEntityTest()
{
var swatch = Stopwatch.StartNew();
NorthwindEntities db = new NorthwindEntities(
ConfigurationManager.ConnectionStrings["NorthwindEntities"].ConnectionString);

for (int orderId = 10248; orderId < 11078; orderId++)
{
var query = from o in db.Orders
where o.OrderID == orderId
select new
{
o.OrderID,
o.OrderDate,
o.Customers.CustomerID,
o.Customers.CompanyName,
ProdCount = o.Order_Details.Count
};

foreach (var item in query)
{
string s = item.CompanyName;
}
}
return swatch.Elapsed;
}
By the way, that original blog post has been criticized by an anonymous guest, who pointed out that my test routines for stored procedures, dynamic SQL and parameterized SQL were not equivalent to LINQ test, because I never read any values from SqlDataReader object after opening it. So, I changed the code in all three methods as follows:
SqlDataReader dr = cmd.ExecuteReader();
while (dr.Read())
{
string s = Convert.ToString(dr["CompanyName"]);
}
dr.Close();
I'm glad to say that this change didn't make any difference to test results: LINQ to SQL is still 40 times slower than stored procedures or dynamic SQL. But Entity Framework results were much worse: 2.2 times slower than LINQ to SQL. Here is the full table:

METHODAVERAGE TIME, ms
Stored Procedure110
Dynamic SQL115
Parameterized Dynamic SQL126
LINQ to SQL4,699
Entity Framework10,547

So, in the end, I rewrote the data layer of the application using LINQ to SQL.
Updated test harness code is available here: http://members.cox.net/rmamedov/blog/EntityFrameworkTestHarness.zip

Saturday, September 26, 2009

Two Strategies For Asynchronous Message Processing

Asynchronous Message Processing

Asynchronous message processing is a common pattern in designing highly available web applications. Essentially, it means that certain tasks are not processed immediately within the scope of a HTTP request. Instead, they are handled asynchronously, by a different application unit, which in Microsoft realm is often (but not always) implemented as a Windows service. This helps streamline the web application and make it appear faster to the end user.

There is a special class of tasks that are suitable for asynchronous message processing, and it is different from typical scheduled jobs (also executed by a separate application service). Usually, they are closely related to user interactions with the main web application, but are too process-expensive or slow to be incorporated into the page processing workflow. Dynamic rendering of a multi-page PDF report is a good example since it falls under both categories. Sometimes, the tasks are not necessarily slow, but they do not produce any information for the user, for example keeping an audit trail of user activity within the system.

Regardless of the specific task being "outsourced", web application exchanges information with the processing service using discrete messages. In a typical design, messages flow only one way: from web application to the service (although there are scenarios where two-way dialog is preferable). There are two main strategies for implementing this design, MSMQ and SQL Server Service Broker.

Technologies

Although I remember the days when you had to buy 3rd party applications (like IBM MQSeries) if you wanted message queuing functionality, Microsoft has incorporated MSMQ 1.0 into Windows NT back in 1994. Fifteen years later, we are now at version 4, and MSMQ is considered to be a proven messaging platform. Even the advent of Windows Communication Foundation did not make it obsolete, since WCF includes special MSMQ binding. Service Broker is a much more recent addition to Microsoft tools lineup; it debuted with SQL Server 2005 and was updated in SQL Server 2008.

Message Size And Content Validation

MSMQ uses special memory-mapped files to store messages, while Service Broker relies on SQL Server database engine for storage. Message size limit is 4Mb and 2048Mb, respectively (Service Broker uses varbinary(max) data type to store messages). When it comes to describing the content of the message, MSMQ offers 18 "Body Type" values, such as integer, double, date, ANSI or Unicode strings, etc. but this information is optional: MSMQ doesn't validate message contents. Service Broker, on another hand, has just 3: empty, XML, and XML with schema, but it does validate the contents and rejects messages that don't conform.

Queue-Server Affinity

When developing high-performance MSMQ applications, one important best practice is "send remotely, receive locally" (here is a link to the article that explains it). What this means is that the process reading from the queue has to exist on the same machine where the queue is located. This impacts overall system scalability, because it makes adding additional processing servers more difficult.

Service Broker doesn't have such limitations, although it does require an instance of SQL Server database engine. In enterprise applications, it is generally a good idea to use a dedicated instance and host all message queues there. Applications that send and receive messages will use this instance as a global hub; they can be physically located on different servers.

Programmability

There are a number of ways to develop MSMQ applications: you can use native API, COM, or .NET. Programmers can use literally any language, from VB to C# to C++. Service Broker is less accessible: its main language is Transact-SQL and in order to use it in, say, a .NET application, you will need to write an ADO.NET wrapper. SQL Server programming samples include one such wrapper, Microsoft.SqlServer.Broker.dll, which exposes classes like SsbService and SsbMessage that shield developers from Transact-SQL.

Thursday, August 13, 2009

Book Review: "The Wikipedia Revolution"

"The Wikipedia Revolution" by Andrew Lih tells the short but enchanting story of Wikipedia: how the project conceived as a fairly regular for-profit web enterprise evolved into something dramatically different. In just a few short years after Nupedia creators embraced wiki-wiki-web and opened the site to everyone with a browser, Wikipedia grew to be one of the top 10 most visited sites on the Internet. It has unparalleled reach with its 259 supported languages, and at least 25 of these languages include over 100,000 articles. All of this has been achieved with essentially all-volunteer force from around the world.

There are some aspects of the book I didn't like very much. For example, it looks like the author has applied one of the Wikipedia editing principles: "No original research". All the information in the book is compiled from publicly available sources; there are no new interviews. Numerous biographies of geeks are rather boring, and most of the additional material about Linux, Free Software Foundation, Mozilla, etc. belongs in sidebars, not main text. (I was reading Chris Anderson's "Free" at the same time and it is amazing how much of that general material is echoed in both books.)

That being said, I am positive you will learn something new from "The Wikipedia Revolution" (unless you are a seasoned wikipedian, of course). Like the intricacies of maintaining three different scripts of the same language (check out Kazakh Wikipedia), for example. Or what roles in the organization are played by administrators and bots. Or what happened when someone googled the word "jew" and didn't like what he saw.

But in my opinion, the most interesting part discusses various controversies surrounding Wikipedia. How does a quality of articles produced by countless anonymous contributors compare with the quality of established encyclopedias, such as Britannica? How does the open system protect itself from vandalism and libel without becoming a closed system? How many articles is too much? (Does every high school need to have an entry? What about elementary schools?) Last but not least, what happened when it was revealed that a prominent Wikipedia contributor and administrator falsely pretended to be a university professor?

Enjoy the book!

Tuesday, July 14, 2009

Exposing EntLib to COM Clients

All of us (well, most of us) know and appreciate the benefits of Enterprise Library (EntLib) Application Blocks: they solve common problems, encapsulate best practices, implement design patterns. They are easy to use and not hard to extend. What's even better, they ensure consistency across different applications and development teams.

Sadly, but all this goodness is only available to .NET code.

I do not have exact statistics, but anecdotal evidence suggests that even software companies firmly committed to .NET platform still have 25-50% of their codebase in C++, VB 6, or some form of VBScript. The ratio will continue to shift but legacy code is unlikely to disappear anytime soon (after all, mainframes and COBOL are still with us). And of course, all that code needs to be maintained.

Programmers are rarely enthusiastic about legacy code maintenance. Part of the reason is that such code is often a reverse of EntLib: it doesn't encapsulate best practices, doesn't use design patterns, is difficult to use. Yet, we cannot afford to rewrite the whole thing and have to be content just patching holes. Therefore, I think many will welcome the possibility to somehow plug in Application Blocks into their legacy code. Parts of EntLib aren't useful in the non-.NET world, of course, but things like logging, caching, and cryptography are perfect integration points. 

In the remaining part of this post, I will describe how this can be done and will use Logging Application Block as an example.

COM Facade
There are many different ways to expose EntLib functionality to older applications, but COM Interop is probably the most efficient. The idea is to use a "facade" design pattern: create a .NET class that will be exposed to COM clients via Interop and pass through calls to EntLib.

I decided to derive this new facade from System.EnterpriseServices.ServicedComponent and host it in a COM+ server application. This way we can take advantage of a couple of very useful services provided by COM+ infrastructure.
[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
public class EntLibAdapter : ServicedComponent, IEntLibAdapter
Interface IEntLibAdapter contains a single method:
[ComVisible(true)]
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
public interface IEntLibAdapter
{
void WriteLog(ref ILogData log, string category, int priority);
}
Here, ILogData is a name of another interface that defines multiple properties collected for logging. Our COM clients will invoke WriteLog method of the facade class and Logging Application Block will determine whether to log or not (based on category and priority), and which logging trace listener to use. This, however, requires that Application Block be properly initialized.

Initialization
EntLib application blocks rely on XML configuration which is usually stored in the web.config or exe.config file. In our scenario, entry point is not a .NET application, so we cannot rely on default behavior. Fortunately, EntLib supports multiple configuration sources: for example, it can consume a stand-alone file and read XML configuration from it. Here is how this is achieved for Logging Application block:
FileConfigurationSource configSrc = new FileConfigurationSource(fileName);
LogWriterFactory factory = new LogWriterFactory(configSrc);
this.LogWriter = factory.Create();
Because the facade class is a ServicedComponent, we can take advantage of the COM+ Activation service. When object is created, COM+ will invoke Construct method and pass a string that in our case will contain full path to the configuration file:
[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
[ConstructionEnabled(true, Default = @"C:\EntLibAdapter.config")]
public class EntLibAdapter : ServicedComponent, IEntLibAdapter
{
protected override void Construct(string s)
{
if (File.Exists(s))
{
FileConfigurationSource configSrc = new FileConfigurationSource(s);
LogWriterFactory factory = new LogWriterFactory(configSrc);
this.LogWriter = factory.Create();
}
}
}
Object Pooling
In a typical usage scenario, EntLibAdapter object will be constructed, WriteLog method invoked, and then the object will be destroyed. Notice that the initialization step is fairly expensive - it involves reading a file from disk, parsing XML, building up objects. We can improve performance and increase overall system scalability by maintaining a pool of objects. That way EntLibAdater instances do not get destroyed after client releases the reference - they simply go back to the pool. Object pooling is another built-in service in COM+, so all we need to do is mark our class to participate:
[
ComVisible(true),
ClassInterface(ClassInterfaceType.None),
ConstructionEnabled(true, Default = @"C:\Nexsure\Installation\Dlls\EntLibAdapter.config"),
EventTrackingEnabled(true),
ObjectPooling(true, MinPoolSize = 5)
]
public class EntLibAdapter : ServicedComponent, IEntLibAdapter
Object pooling may not be the best approach if you plan to use flat file logging. In the example above, there will always be 5 instances in the pool, and each will create its own log file (4 of them will have a GUID-based file name). This should not be a problem if your primary target is database or Windows Event Log.

Deployment
Since EntLibAdapter is a ServicedComponent, it has to be strongly named and registered using RegSvcs.exe. In addition, EntLib assemblies that it depends on, such as Microsoft.Practices.EnterpriseLibrary.Logging.dll, need to be placed in the GAC.

Thursday, June 04, 2009

Design For Operations, Part III - MMC Integration

Back in 2006 I started writing about designing applications with IT department in mind. I'm glad to say that my position didn't change since then - I still believe computer software should be friendly to its end users, which are either customers (mobile, desktop and web apps) or IT engineers (web apps and services). Unfortunately, these two groups are not represented equally in the design process: business analyst is a voice of the customer, but who is a voice of IT?

MMC Background
In my previous posts I covered such aspects as event logging, performance counters, and WMI integration. Today I wanted to discuss Microsoft Management Console (MMC) which was originally developed for Windows NT Option Pack. The idea was to create a common interface for managing IIS, Certificate Server, and Transaction Server, so that administrators have fewer tools to learn. This was further extended in the next release, MMC 2.0 (included with Windows XP/Server 2003), when a concept of the snap-in was added. Snap-in was a COM in-process server that MMC would communicate with, thus allowing any third-party application to have a custom management screen within MMC. In order to develop custom snap-ins, you would need to be well-versed in C++ COM development, as there were 30-something interfaces that could be used. Development of snap-ins in managed code was not supported, although there were custom frameworks, for example, an open-source project called MMC.NET.

MMC 3.0, which was shipped with Vista and Server 2008 (but available for download for older platforms), includes a managed layer and thus natively supports snap-in development in any .NET language. Everyone but hard-core C++ programmers would agree that this can be done faster, with fewer lines of code and simplified maintenance. And even they will have to admit that the ability to use WinForms inside MMC is really cool.

Setting Up Solution
In order to develop our custom MMC snap-in using C# 2008, we will create a class library project. However, before you begin, there is one important step: executing %WINDIR%\System32\MMCPerf.exe from the command prompt. This will put MMC assemblies in the GAC and NGEN them. After creating new project, add a reference to Microsoft.ManagementConsole.dll by browsing to the following folder: \Program Files\Reference Assemblies\Microsoft\mmc\v3.0\. Unfortunately, the assembly doesn't appear in the .NET tab of the "Add Reference" dialog.

Writing Code
We will start by adding two classes to the project: one derived from Microsoft.ManagementConsole.SnapInInstaller, will be used to register custom snap-in with MMC, and another, derived from Microsoft.ManagementConsole.SnapIn, will serve as an entry point.

using System.ComponentModel;
using System.Security.Permissions;
using Microsoft.ManagementConsole;

[assembly: PermissionSetAttribute(SecurityAction.RequestMinimum, Unrestricted = true)]

namespace Microsoft.ManagementConsole.Samples
{
[RunInstaller(true)]
public class InstallUtilSupport : SnapInInstaller
{
}

[SnapInSettings("{9627F1F3-A6D2-4cf8-90A2-10F85A7A4EE7}",
DisplayName = "- Sample SnapIn",
Vendor = "My Company",
Description = "Shows FormView")]
public class SelectionFormViewSnapIn : SnapIn
{
}
}
Note the attribute that decorates SelectionFormViewSnapIn class. All snap-ins are defined in the  Registry, so we have to provide a GUID, and also specify metadata which will be displayed in the catalog. We can leave the body of the installer empty, but SnapIn class requires a constructor.

public SelectionFormViewSnapIn()
{
// Create the root node.
this.RootNode = new ScopeNode();
this.RootNode.DisplayName = "Selection (FormView) Sample";

// Create a form view for the root node.
FormViewDescription fvd = new FormViewDescription();
fvd.DisplayName = "Users (FormView)";
fvd.ViewType = typeof(FormView);
fvd.ControlType = typeof(SelectionControl);

// Attach the view to the root node.
this.RootNode.ViewDescriptions.Add(fvd);
this.RootNode.ViewDescriptions.DefaultIndex = 0;
}
In the constructor, we are effectively defining the root node of the snap-in. In this example, we will be using a WinForms user control called SelectionControl. This is a regular UserControl that implements a special interface Microsoft.ManagementConsole.IFormViewControl, which really only has a single method: void Initialize(FormView view). This is where we would put our initialization logic.

Deployment
After successful build of the solution, we will end up with a single DLL. Deploying it is very straightforward: all you need to do is execute InstallUtil.exe against it. Assuming you didn't skip the first step (running MMCPerf.exe), you should see no errors. Start MMC and choose "Add/Remove Snap-in" from the File menu, then click the "Add" button. You should see your custom snap-in appear in the list.

Sunday, March 01, 2009

How To Force Partial Postback

ASP.NET UpdatePanel control allows us to program web pages with partial postback. The page still goes through most of its lifecycle stages, but only a portion of HTML is updated in the browser. This is a quick and efficient way to improve user experience by reducing page flicker.

Usually, the events that trigger partial postback of an UpdatePanel are defined declaratively in .aspx file. For example, events by child controls are considered triggers by default. If a control is located outside UpdatePanel, it can still be declared a trigger using markup:

<asp:UpdatePanel ID="MyUpdatePanel" runat="Server">
<Triggers>
<asp:AsyncPostBackTrigger ControlID="Button1" />
</Triggers>
</asp:UpdatePanel>

However, sometimes it is necessary to trigger partial postback programmatically. For example, consider an UpdatePanel that is used together with CollapsiblePanelExtender. We want to force postback when user expands the panel. The logical place to do so is inside the "Expanded" event handler, and the code is very simple:

function onCollapsiblePanelExpanded(sender, args) {
__doPostBack('<%= MyUpdatePanel.ClientID %>', '');
}

If we need specific server-side logic to handle partial postback, ScriptManager control offers two properties: bool IsInAsyncPostBack and string AsyncPostBackSourceElementID. The former indicates whether page is processing a partial postback, and the latter contains ID of the UpdatePanel being processed.

Friday, February 13, 2009

Using Web Client Software Factory With Mobile Web Forms

After I somewhat successfully solved the problem Visual Studio 2008 has with mobile web forms, I had to tackle another challenge. The website is built with Web Client Software Factory, a powerful and flexible ASP.NET framework from Microsoft Patterns & Practices team. WCSF, or, more specifically, CWAB (composite web application block) provides dependency injection to the application. Unfortunately, the most recent release of WCSF doesn't include any support for mobile web forms. Of course, this release is almost one year old, and I know that good people of P&P are planning a new release for 2010, so hopefully this will be addressed, but in the meantime here is the solution I came up with.

WCSF guidance package includes a nice set of recipes that automate creation of web forms, master pages and user controls. The boilerplate code that is autogenerated for code-behind classes defines them as derived from Microsoft.Practices.CompositeWeb.Web.UI.Page class instead of standard System.Web.UI.Page:

public partial class MySummaryView : Microsoft.Practices.CompositeWeb.Web.UI.Page, IMySummaryView
{
}

The Page class overrides the OnPreInit method, and that is where dependency injection "magic" happens:

protected override void OnPreInit(EventArgs e)
{
base.OnPreInit(e);
Microsoft.Practices.CompositeWeb.WebClientApplication.BuildItemWithCurrentContext(this);
}
We can use similar approach for mobile web forms. Of course, there is no class in WCSF that is derived from System.Web.UI.MobileControls.MobilePage, so we will have to create our own base class in App_Code (or in a shared class library):

namespace Microsoft.Practices.CompositeWeb.Web.UI
{
public class MobilePage : System.Web.UI.MobileControls.MobilePage
{
protected override void OnPreInit(EventArgs e)
{
base.OnPreInit(e);
Microsoft.Practices.CompositeWeb.WebClientApplication.BuildItemWithCurrentContext(this);
}
}
}
Now, as long as we derive our mobile web form from this class, we can declare a Presenter property and let CWAB inject it at runtime. There is one drawback, though: we still need to manually create the presenter class and view interface, something that the guidance package recipe used to do automatically. Unfortunately, I don't know GAT well enough to create my own recipe for mobile web form, so the workaround I am using is this:
  1. Execute "Add page with presenter" recipe
  2. Modify .aspx file to register "mobile" tag prefix
  3. Modify code-behind file to change the base class

Friday, February 06, 2009

Mobile Web Forms in Visual Studio 2008

I recently discovered that Visual Studio 2008 dropped support for ASP.NET mobile. You can create ASP.NET websites, of course, but try adding a mobile web form or mobile user control - these item templates are no longer there.

Omar Khan wrote a post for Visual Web Developer Team blog which describes a workaround but doesn't explain why this happened in the first place. One big problem with that workaround is that you can't download it due to a broken link.

I found that one way to solve the problem is to copy mobile item templates from Visual Studio 2005 (assuming you still have it installed) to a special folder where VS 2008 will look for user item templates. Here's the detailed how-to:

1) Find item templates in VS 2005 folder:


2) Find user item templates folder:



3) Copy MobileWebForm.zip, MobileWebUserControl.zip, and MobileWebConfig.zip to that folder

4) Restart VS 2008. Mobile web items now appear in the "Add New Item" dialog under "My Templates":



One problem still remains: VS 2008 designer doesn't display mobile forms and controls. This isn't a major issue for me because I hardly ever use the designer.

Sunday, January 25, 2009

Cloud Storage or SDS?

Ever since Ray Ozzie has announced Windows Azure on last year's PDC (watch the keynote) there's been a lot of buzz about new platform. Every recent Microsoft event, it seems, included a session or two on Azure. SoCal Code Camp that took place this past weekend at CalState Fullerton had an entire track of cloud-related presentations. 

One general observation: details of the new platform are still, well, cloudy. Windows Azure is presently in a CTP stage; nobody expects an RTM until the end of 2009. There are a lot of technology-, process-, and cost-related questions that no one yet knows the answers to. What's worse, marketing geniuses at Microsoft decided to slap Azure label on a set of technologies that originated in different parts of the company (and even Microsoft evangelists admit that there is very little coordination).

Take a look at the obligatory Azure platform stack slide:
My initial assumption about SQL Server Data Services (SDS) was that it is somehow built on top of Azure. Apparently, SDS is a completely separate service. In fact, you don't even need to have an Azure application in order to use it.

Let's take a closer look at the data support for cloud applications. This is, in my opinion, the biggest paradigm shift for developers and architects. After all, it's easy to understand the concept of deploying your application code to a whole bunch of virtual servers, but how are we going to survive without our beloved connection strings, stored procedures, triggers?

There are two options available to us, cloud storage and SDS. Both are going to be reliable, scalable, highly available, and support terabytes of data. On the back end, both will utilize a vast  network of SQL Server nodes that use some advanced algorithms to support distributed data storage and replication. Below is a side-by-side comparison.

Signup. When you sign up for Azure, you receive a separate storage account. To use SDS you will need to get yet another account.

Hierarchy
Cloud Storage: Provides account/container/entity model for your data.
SDS: The model is similar - authority/container/entity

Data Abstractions
Cloud Storage: Supports blobs (basically, named files with metadata) up to 50Gb, tables (which are essentially lists of entities, not database tables), and queues with message size up to 8Kb.
SDS: Only works with entities, which are similar to the tables above.

Data Access
Cloud Storage: Blobs and queues can be accessed via REST, but tables are also exposed via ADO.NET Data Services. This allows for a more convenient API (for example, you can query a table using LINQ). Large blobs can be uploaded by small 4Mb-sized chunks.
SDS: Entities can be queried using REST. Although there is no ADO.NET Data Services support, you can pass a LINQ-style query in the HTTP request (query may even join entities).

*** UPDATE (March 17, 2009)
Someone at Microsoft have finally noticed the striking similarities between Cloud Storage and SDS. Data Platform Insider blog is announcing the change to SDS architecture: REST-based interface will be decommissioned and replaced by a service protocol based on Tabular Data Stream (which has been a SQL Server network protocol since SQL 2000). Public CTP of the new architecture will be available in the middle of this year.