This is good advice, and straightforward to implement under IIS7, but can leave you with a bit of a problem – once you’ve released your site with all of your scripts and stylesheets set to expire next millenium, you can no longer edit them as you please. If you change them and release them with the same name, returning visitors to your site will not see the changes – they will use the cached version that you told them was good for another 1000 years.

Instead, if you need to change one of your files, you need to give it a new name so that clients who downloaded and cached the original version are sure to get the update when it’s deployed. This means you need to update any references to point to the new file, which has the potential to be a bit of a management nightmare.

Obviously, if you’ve only got one or two of each type of file and you’ve referenced them from a master page or similar shared component then you can handle this manually. However, if you’ve got more than that, or if you’re following the recommendation to combine and pack files, then life can get more fiddly.

On a previous project, I’ve used the Aptimize Website Accelerator, which handles all this stuff for you in an extremely slick way. However, for projects that don’t use this, I wanted to come up with a way of making the process easier.

In order to do this I decided to take advantage of ASP.NET’s internal URL rewriting feature, exposed via the HttpContext.RewritePath method. What I ended up with – which you can download from Codeplex if you’d like to give it a try – has two parts, the logic to add and remove version numbers from file names and an HttpModule to apply this logic to incoming Urls. The idea is that the code writing out links to the JS/CSS files can pass those filenames through the provider to add a version number, and the HttpModule will intercept incoming requests for those files and rewrite them to point to the correct filename.

The resourcing versioning provider

Whilst I’m not a massive fan of the ASP.NET provider model, I’ve recently been feeling the pain of using a variety of different open source projects with overlapping dependencies on different versions of the same tools. I therefore decided to use the provider model to allow me to swap out different ways of generating the version number.

The first important class here is ResourceVersioningProvider. This defines the two methods that providers need to implement.

It should be pretty obvious what they do from their names.

I’ve then got a base class,  ResourceVersioningProviderBase which contains the code that’s independent of how the version number is generated (basically the filename parsing/modifying logic as well as some caching.) This class is abstract and defines a single abstract method, GetVersionNumber(). This should return the version number that’s going to be used to make the filename unique.

The final relevent class is AssemblyBasedResourceVersioningProvider, which inherits ResourceVersioningProviderBase and implements GetVersionNumber by returning the full version number of an assembly specified in the configuration. The configuration itself looks like this:

As you can see, there’s also an attribute that controls which file extensions the provider will add version numbers to. This means that if you call “AddVersionToFileName” and pass in an a filename with a different extension, no version number will be added.

The resource versioning HttpModule

The HttpModule that forms the other half of the solution is pretty straightforward. Every incoming Url is cracked open using the Uri.Segments property, and the final segment (assumed to be the requested file name) is passed into the provider to remove the version number, if present. If the provider returns a filename that differs from the one in the request url, it is used to build a new Url that can be passed to HttpContext.RewritePath. Processed Urls are cached (regardless of whether or not they require a rewrite) to improve performance.

In Practice

The module is in use on the Who Can Help Me? app I introduced in my previous post. If you have a look at the source for the homepage, you’ll see that all the CSS, JavaScript and DotLess files have the current WhoCanHelpMe.Web assembly version number as part of their name. You can get at the Who Can Help Me? source via it’s Codeplex site to see how it’s implemented.

An Alternative Approach

Despite having written this post a month or so ago, I was only spurred into posting it by reading about this alternative approach from Karl Seguin. He’s got an interesting series going on at the moment covering ASP.NET performance – it’s well worth a read. I can see that it would be fairly straightforward to replace my AssemblyBasedResourceVersioningProvider with a FileHashBasedResourceVersioningProvider, although given that no. 1 on my to do list at the moment is learning  NHibernate, it might be a while before I get round to it!

As I mentioned above, the compiled assemblies and the source code are available from my Codeplex site. Please have a play and let me know what you think.

@jon_george1

This is the final part of a series of posts on optimisation work we carried out on my last project, www.fancydressoutfitters.co.uk – an ASP.NET MVC web site built using S#arp Architecture, NHibernate, the Spark view engine and Solr. There’s not much point starting here – please have a look at parts 1, 2, 3 and 4, as well as my post on improving YSlow scores for IIS7 sites, for the full picture.

In the posts on this series, I’ve reflected the separation of concerns inherent in ASP.NET MVC applications by talking about how we optimised each layer of the application independently. Good separation of concerns is by no means unique to applications built using the MVC pattern, but what stood out for me as I became familiar with the project was that for the first time it seemed like I hardly had to think to achieve it, because it’s so baked into the framework. I know I share this feeling with Howard and James (respectively architect and developer on the project), who’ve both talked about it in their own blogs.

The MVC pattern also makes it much easier to apply optimisations in the code. For example, it’s much easier to identify the points where caching will be effective, as the Model-View-ViewModel pattern makes it straightforward to apply a simple and highly effective caching pattern within the controllers. I know that this kind of thing isn’t limited to performance work – for example, our team security guru certainly felt that it was easier to carry out his threat modelling for this site than it would have been in a WebForms equivalent.

On the flip side, this process also brought home to me some of the dangers of using NHibernate. It’s an absolutely awesome product, and has totally converted me to the use of an ORM (be it NHib or Entity Framework). However, the relatively high learning curve and the fact that most of the setup was done before I joined the project made it easy for me to ignore what it was doing under the covers and code away against my domain objects in a state of blissful ignorance. Obviously this is not a brilliant idea, and properly getting to grips with NH it now jostling for first place on my to-do list (up against PostSharp 2 and ASP.NET MVC 2.0, amongst other things.)

My big challenge for future projects is ensuring that the optimisations I’ve talked about are baked in from the start instead of being bolted on at the end. The problem with this is that it’s not always clear where to stop. The goal of optimising the site is to get it to the point where it performs as we need it to, not to get it to the point where we can’t optimise any more. The process of optimisation is one of diminishing returns, so it’s essential to cover issues you know need to be covered and to then use testing tools to uncover any further areas to work on.

That said, in an ideal world I’d like to be able to build performance tests early and use them to benchmark pages on a regular basis. Assuming you work in short iterations, this can be done on an iteration by iteration basis, with results feeding into the plan for the next iteration. My next series of posts will be on performance and load testing, and as well as covering what we did for this project I will be looking at ways of building these processes into the core engineering practices of a project.

Was it all worth it?

I’ll be talking separately about the performance and load testing we carried out on the site prior to go live, but in order to put these posts into some context I thought it might be interesting to include some final numbers. For our soak testing, we built a load profile based on 6 user journeys through the site:

• Homepage: 20% of total concurrent user load
• Browse (Home -> Category Landing -> Category Listing -> Product): 30%
• Search (Home -> Search Results): 30%
• News (Home -> News list -> News story): 10%
• Static Pages (Home -> Static page): 5%
• Checkout (As for browse journey, then -> Add to basket -> View Basket -> Checkout): 5%

With a random think time of 8 – 12 seconds between each step of each journey, we demonstrated that each of the web servers in the farm could sustainably support 1000 concurrent users and generate 90 pages per second. Given the hardware in question, this far exceeds any project I’ve worked on recently.

In the end, we put www.fancydressoutfitters.co.uk live in the run up to Halloween, the busiest time of the year for the fancy dress industry. We did this with no late nights and enough confidence to go to the pub for a celebratory pint within the hour. It was also interesting that the majority of technical colleagues who responded to our go-live announcement commented on how fast it runs (which given the machinations of our corporate network’s internet routing is even more remarkable.) And best of all, we’ve had no major shocks since the site went live.

A final note

If you’ve read this series of posts, I hope you’ve got something out of it. I’d certainly be interested in any feedback that you might have – as always, please feel free to leave a comment or contact me on Twitter.  In addition, the EMC Consulting blog site has been nominated in the Computer Weekly IT Blog Awards 2009, under the “Corporate/Large Enterprise” category – please consider voting for us.

I’d also like to extend a final thanks to Howard for proof reading the first draft of these posts and giving me valuable feedback, as well as for actually doing a lot of the work I’ve talked about here.

@jon_george1

This is the penultimate part in a series of posts on optimisation work we carried out on my last project, www.fancydressoutfitters.co.uk – an ASP.NET MVC web site built using S#arp Architecture, NHibernate, the Spark view engine and Solr. Please also take the time to read parts 1, 2 and 3 for the full picture.

Output caching is something I’ve used extensively on previous ASP.NET WebForm projects, and I was surprised when I found out that it’s relatively immature in ASP.NET MVC. The core library contains the OutputCache filter, which does the same under the covers as the @OutputCache declaration in ASP.NET Web Forms. However, caching a full page is something that rarely suits in the current climate of dynamic and personalised websites – for example (and in common with most ecommerce websites) our site includes a summary of the user’s basket on every page. In the world of web forms, I’d use the asp:substitution control to implement donut caching, or implement fragment caching. Unfortunately neither of those options exist in the brave new world of ASP.NET MVC (if you’re a Java/Ruby/etc developer… stop sniggering, we’re catching you up )

So there are a few ways round this that I’m aware of. The first involves using the partial views to do something similar to the old style ASP.NET Fragment Caching approach. In the WebForms world, this is implemented by splitting your page up into multiple user controls, and applying output caching as appropriate to the individual controls. Here’s a pretty basic example of how a product page on an ecommerce website might be laid out.

The boxes coloured blue, green and purple can all be cached – albeit with different cache lifetimes and keys. The red box, which displays my user  basket total is unlikely to be cachable – at least not at the rendering level. Using the classic ASP.NET fragment caching approach, each of my coloured boxes would be implemented as a separate user control with appropriate OutputCache directives set. The ASPX page itself would contain very little other than the instances of these controls.

So how does this translate into MVC? It turns out that it’s pretty much the same. You create a number of partial views and add an OutputCache directive to their markup. Phil Haack covers the approach in this blog post. There’s just one downside: that approach only works with the standard WebForms view engine, and we’re using Spark, so we can’t go down this route.

I did hear a suggestion that making use of the RenderAction method from ASP.NET MVC 1.0 Futures could be the way forward. Essentially, each of my coloured boxes from the diagram would end up corresponding to a separate controller action, each of which would have an OutputCache action filter applied. These would then be pulled together by a “dumb” controller action whose corresponding view had multiple calls to Html.RenderAction to compose the chunks of the view together.

On the face of it – and assuming you were willing to accept the overhead involved in repeatedly invoking the full MVC request pipeline – it would work. However, there has been mention of a bug with MVC 1.0 which causes the OutputCache filter on an action method called with RenderAction to be applied to the overall request, not just the chunk being dealt with. This kind of thing is probably why the RenderAction method was bumped into MVC Futures instead of being shipped as part of the core MVC 1.0 release.

Phil Haack blogged something else that didn’t quite make it to MVC 1.0 which looked good on the surface. Essentially, it’s an HtmlHelper extension that hooks into the API used by the asp:substitution control. However, I had a bit of a headache in trying to work out how to use it. The problem is that within the callback you supply to the Substitute method, you don’t have access to your ViewData – not a massive surprised once you understand how the post-cache substitution mechanism in ASP.NET works. This means that you need to code some other method – which is going to be a static method you’ve stashed alongside your controllers – that will do all the necessary work, pulling bits of data out of the supplied HttpContext and returning a string to be dumped directly into the view.

There’s no doubt that this would work, and with some thought could be done without losing the testability and separation of concerns that makes the MVC pattern great. However, it’s not an ideal approach for me because it does mean that the pattern and conventions are broken to support what’s essentially an optimisation step. Because of this it will make it harder for people to see what’s going on in the code. I’ve already covered the caching we applied in the service layer; to me, output caching should be contained within the View Engine layer and should not leak beyond that. After all, there’s nothing else in my controller layer or below that couples my code to Spark, so I have no desire to introduce something that does.

Fortunately it turns out that the 1.1 version of the Spark view engine will contain pretty comprehensive output caching. This isn’t part of the v1.0 release, but has been in the development builds for several months now and is stable. It’s absolutely perfect for what I wanted as it allowed me to implement output caching with very few changes outside the views.

Unlike the ASP.NET WebForms fragment caching approach, you aren’t required to split your view into partials – you simply add <cache> elements around the sections of the page you want to cache. These sections can be dependent on different keys and cached for different durations, and there’s also a signalling mechanism that allows cached content to be expired on demand. In our case, we had already tackled the issue of cache keys for a ViewModels when we looked at caching in the controller layer, so it was a simple matter to use these same cache keys to control our output caching.

Spark also contains something called the ValueHolder which effectively allows you to defer the gathering of model data until it’s needed. This means that rather than build up the model for every request, only to pass it to a view which doesn’t need it because it’s entirely output cached, you can build your model using ValueHolder objects containing lambda functions that will only be executed if the data is needed. This seems like an interesting approach, but it’s not one I explored in detail because the caching we’d already implemented on the controllers made it less relevant.

One of my major hopes, which was unfortunately not realised, was that we’d be able to connect Spark’s output caching service to our distributed cache, Velocity. This would further reduce the workload across the web farm because it would mean that once a page was served from one webhead, it would be available pre-built to all of the others. However the output caching mechanism in Spark places unserializable objects into the cache, making it difficult to use with an out-of-process caching mechanism. This is a shame but by no means a deal breaker.

I’ve seen a few discussions around the value of output caching in the MVC world, with some saying that because the views are essentially just writing text to a stream, there is little gain to be had from caching. On a purely subjective level, the output caching did seem to make the site faster. It’s difficult to be sure because there is no way of enabling/disabling output caching via config in Spark, so it’s not easy to do comparative tests in a realistic environment. I can see the argument, and I’d certainly agree that out of the different focus areas output caching made the least difference to overall performance, but I believe it did make a difference and for the minimal effort involved in implementing it, was worth it.

In the final section, I’ll talk about my views on optimising this MVC based site compared to my experiences in the WebForms world, and share some hard numbers gained from our performance testing.

@jon_george1

This is part 3 in a series of posts on optimisation work we carried out on my current project, www.fancydressoutfitters.co.uk – an ASP.NET MVC web site built using S#arp Architecture, NHibernate, the Spark view engine and Solr. Please see  part 1 and part 2 for the background.

I doubt many people will disagree with me when I say that one sure-fire way of making a piece of code run faster and consume less of your precious processing power is to do less work in that piece of code. Optimising your code is part of the answer, but however quickly your pages run when it’s just you on the site you will struggle to serve large numbers of users unless you have a good caching strategy in place.

There are a number of places that caching can be applied in an application built using S#arp Architecture. Here are a few:

• You can use the NHibernate second-level cache to store the results of database queries.
• You can apply caching wherever it’s appropriate in your model layer.
• You can cache the results of controller actions; and
• You can apply output caching to the rendered views (to be covered in part 4).

However the whole thing is something of a balancing act. You have to understand how often things are going change and weigh up the benefits of longer caching durations against the need to reflect updates in a timely fashion. You need to understand how your platform architecture affects your caching strategy – for example, what happens if you have multiple web servers each maintaining it’s own local cache? You also need to make sure that once you’ve decided on an approach, people stick to it. I’ve never been a big fan of using the standard HTTP Cache directly – it requires you to specify the caching interval every time you add an object to the cache. Even if you implement some constants to hold the intervals, you still run the risk of people choosing the wrong one.

How we approach caching

Both my current S#arp Architecture-based project and my previous WebForms one make extensive use of Dependency Injection to decouple our assemblies and increase testability. It follows that we needed to create an abstraction for the cache anyway, so we took the opportunity to kill two birds with one stone and introduce a more convention-based approach to the cache. James has recently blogged about the benefits of removing decisions from our day-to-day coding activities, and this is another example. In our abstraction, we treat the cache as a set of named areas and assign cache duration to those areas instead of to the individual objects. A few examples of the named areas we use are:

• Content – medium lifespan, intended for content pulled from the CMS;
• Resources – very long lifespan, containing values from our SQL-based resource provider (used instead of the standard RESX file-based provider); and
• Site Furniture – long lifespan, containing less frequently changing data used in the site header and footer.

(Note: If this approach interests you at all, let me know – the specifics of the interface and implementation are two much detail to include here, but I can cover them in a separate post if there is interest.)

A nice bonus from using this abstraction on the current project was that it allowed us to use the current CTP of Velocity, Microsoft’s new distributed cache. The abstraction – together with our use of the Windsor IoC container – provided us with a handy safety net: if at any point we found an issue with using Velocity, switching back to the standard HTTP cache would be a simple configuration change. As I’ll cover in a future post our soak testing showed that Velocity is fast and stable, but even post go-live, it would still be a simple matter to make the switch if necessary. One of the tenets of lean development is that you should delay decision making until the last responsible moment – for us, the last responsible moment for deciding on a caching solution could be deferred until go-live, by which point we’d been using Velocity for around 5 months.

Applying caching in the controllers

Our first area of focus was to look at caching the results of controller actions. One of the great things about ASP.NET MVC is that things are so much simpler than in the web forms world (check this out for a pretty picture of the request handling pipeline). It’s far easier to understand where you do and don’t need to apply caching in your application code, and we realised that the most effective thing to do was to put the majority of ours in the controllers.

Howard posted a diagram of our architecture on his recent blog post about AutoMapper. From this you might be able to infer that our controllers tend to follow a pretty simple pattern:

1. Input view model is populated from request data by a model binder.
2. Input view model is mapped to domain objects.
3. Application services layer is used to do work with those domain objects.
4. Resulting domain objects are translated back into a view model that can be passed to the view.
5. Appropriate ActionResult is returned.

This is the most complex scenario – not all steps are required for every action.

In an ideal world we’d be caching the ActionResults returned by the controllers but this isn’t something we can do because they aren’t serializable and therefore can’t be cached in Velocity. We therefore have to settle for caching the ViewModel for each action, which gives us this pattern:

The [Cached] attribute is actually a PostSharp aspect that caches the result of the method call, and I have both good intentions and half finished blog posts on this subject.

If the action method takes parameters, these are passed straight through to the ActionInner method – we do the minimum amount of work possible in the action method itself.

Dealing with changing content in the CMS

In an ideal world, we would set most of the caches to have an extremely long life, then implement a mechanism to allow individual caches to be flushed as and when required. Anyone who has ever used Microsoft Commerce Server will be familiar with the SiteCacheRefresh handler which allows caches to be cleared in response to a web request. However we encountered an issue here: the current release of Velocity (CTP3), does not support enumeration or flushing of named caches – these operations are limited to cache regions. The downside with cache regions is that they are tied to a single server, which pretty much removes the benefits of using a distributed cache.

As I’ve previously mentioned, our system uses N2 for Content Management and the majority of our pages are represented in the CMS. We therefore cached the ViewModel for each page using a key based on the unique ID of the page and the last updated date of the page. This works well because it means that pages can be cached indefinitely, but as soon as someone makes a change to the page then a new version will be created and cached. Obviously this means that the old version will be lingering in the background – the downside to this approach is that you may well fill up the cache more quickly and require Velocity to evict more frequently.

This approach isn’t perfect – it doesn’t cover the product pages, and doesn’t take care of changes in the site hierarchy that would mean we need to regenerate the navigation.  As a result, we’ve implemented an additional hack: all cache keys used for content pages include the last updated time of a single page within the site. Touching that page will cause every cache key used at the controller level to change. I don’t particularly like it, but it does work.

The longer term plan would be to look at this again as and when Velocity moves on. With any luck, the next CTP (now overdue) will make some changes in this area. The ideal plan would be to:

• Hook into N2’s persistence mechanism – from reading the documentation and looking at the code, it should be possible to receive a notification when things change.  This could then be used to remove stale objects from the cache, or flush an entire named area as required.
• Implement a restricted controller that allows clearing of individual caches (in the style of the CS SiteCacheRefresh handler mentioned above.) This would be useful if we needed to clear a cache in response to an external action – such as an ETL process pulling data from a back office system.

Caching elsewhere in the application

Once we’d implemented the caching in the controllers and configured the NHibernate second-level cache in both our custom code and N2, there were very few places in our model layer that we needed to apply caching. Once again we hit the point of diminishing returns – we could spend time profiling the application and identifying any remaining bottlenecks, or we could move on and look at what we expected to be the final big win: output caching, which I’ll be talking about in the next post of this series.

A final word about caching

On a lot of projects I’ve worked on, caching is something that tends to be implemented at the end. There are advantages to this – it can help to avoid premature optimisation, and you will (hopefully) have a good understanding of the moving parts in your codebase. However, there is one major downside that anyone who’s done this will be familiar with: once you check in the changes, every single bug that is raised for the application will be blamed on your caching. Accept it, live with it, and be prepared to be known as a) grumpy, and b) easy to wind up. Just take satisfaction in the knowledge that when you get it right, you will be setting yourself up for success when you finally put your site live.

As always, I’d appreciate feedback on these posts – either by leaving a comment, dropping me an email or via Twitter.

@jon_george1

This is part 2 in a series of posts on optimisation work we carried out on my current project – an ASP.NET MVC web site built using S#arp Architecture, NHibernate, the Spark view engine and Solr. Please see “Optimising an ASP.NET MVC web site part 1 – Introduction” for the background.

Whilst there are many different things you can do to a web application to kill performance, having a badly implemented database – or a well implemented database that you access in a bad way – has got to be right up there with the best. We therefore made the database our first port of call, and started off with a pretty simple approach – fire up  SQL Server Profiler, start a new trace and see what the pain points are in each database.

Using SQL Server Profiler to find data access mistakes

As I mentioned in the introduction, the goal of our optimisation wasn’t to do everything, just to get the main things we thought were causing problems before we began our load testing. If you use the TSQL_Duration template in SQL Profiler for your trace, you can hit your site and quickly see what the most expensive queries are.

The screenshot above shows the results of repeatedly executing three different stored procedures in a random order. As you can see, the results are organised in ascending order of cost, with the interesting column being “Duration” – which shows the execution time in milliseonds.

When Howard first ran this against some of our landing and search result pages he quickly noticed a large nugget of smelliness at the bottom of the list, in the form of repeated calls to a stored procedure that was taking a whopping 400ms to execute. Whilst 400ms is not a long time – I’m sure it’s taken longer than that for you to read this sentence so far – when you call it 20 times you’ve suddenly spent 8 seconds just on database calls before any other page activity.

Digging into this identified three separate, but related issues.

1. One of our NHibernate mappings included a view based on a recursive query

Of course, this was something we’d told NHibernate to do. Our entities are organised hierarchically, and an entity can appear in multiple positions in the hierarchy. When we pass entity data to Solr for indexing, we work out the full list of paths for an entity and pass that through (see item 3 for more details.) This was done by creating a CTE to generate the list of hierarchy paths for each entity.

For the uninitiated, a Common Table Expression (or CTE) is a T-SQL construct that (amongst other things) enables you to execute recursive queries. They are very handy when dealing with hierarchical datasets, but aren’t the most efficient queries you can execute. Some time after the initial creation of the view, we’d realised that it would be useful to have the data it contained as part of our entity so we added a mapping into our NHibernate configuration. This meant that accessing that data would cause NHibernate to go off and execute a SELECT statement which included a join from the main entity table to the view. This query took in the region of 400ms.

We have two main types of page on the site landing pages and search/browse pages. The landing pages were causing this query to be executed 13 times and the browse pages were causing it to be executed 20 times, so it’s no wonder that performance had dropped. Whilst the view was never intended for use in this way, the requirement to have the data as part of our entity was still valid.

The simple solution to the problem was essentially to materialize the view. SQL Server can do this by turning the view into an indexed view – adding a unique clustered index to it does this. However, this approach isn’t applicable when the SELECT statement for the view uses a CTE, so we went with a more basic approach – since our product catalogue is actually managed in the back office and populated via ETL, we replaced the view with a table (complete with all the appropriate indexes) and tweaked the ETL to populate this table at the same time as all the rest.

For the pages in question, load time dropped from around 8 seconds to under 2, at which point we all got quite excited. However, this wasn’t solely to do with the materialisation of the view, as the investigation also showed up that…

2. Everything was being lazy loaded

By default, NHibernate uses lazy loading across the board. Depending on your object model and how it is used, this can lead to massive inefficiences. The classic example is the "SELECT N+1” anti-pattern, in which you retrieve an entity from a repository then iterate over a collection on that entity. If you’ve left NHibernate lazy loading the values, then this results in a SELECT statement being issued for every iteration of the loop. Have a look at this page on the NHibernate Profiler site for more information.

Howard spent some time using the NHibernate profiler to better understand what our app was doing, and made a number of tweaks to our mapping configuration to apply eager loading where it made sense. This provided another significant improvement, dramatically reducing the number of database calls made by the application.

3. An architectural rule imposed for performance reasons had been broken

We’d made an architectural decision to drive the search and browse functions on our website exclusively from Solr. We did this because it gives us more options for scalability. Since all the data in question comes from a single database, pulling the data directly out of that database would mean that as website load increases then so does the load on the database. The problem with that is that it’s difficult to scale a single database out – you can throw CPUs and RAM at your database server, but you’re always going to hit a limit, at which point you face some architectural challenges. This kind of basic architecture is shown in the following diagram:

Even though the main entity database is static, meaning that it would be possible to have a number of replicas of this database (potentially even one per webhead), this would require architectural change and would bring with it a new set of issues around data consistency. By pushing that load onto Solr, which has a far more straightforward scale-out story, we can grow far further without requiring a redesign. Solr basically stores a flat set of data in a form optimised for searching, and provides access via a web service. This means it is straightforward to have multiple instances of Solr running behind a load balancer. Solr makes this kind of setup even easier, as it supports a master-slave configuration as shown in the following diagram (I apologise now for the proliferation of connecting lines – I’m more of a reluctant architect than a Powerpoint architect):

In this example, the web tier will still talk direct to the data tier for some tasks – it’s unavoidable. However, we introduce the search tier which consists of a set of load balanced search servers, each of which contains an identical copy of the search index. In order to build that index, we push data from the database into the Solr master server, and the Solr master server indexes it and pushes the result out to each slave. If you can see past the nasty mess of lines, it should be obvious that as load grows, adding more webheads and/or Solr slaves is a relatively trivial operation.

However, you can have the best intentions in the world when you design your solution, but if you then go on to break the rules then it’s not going to happen. In our case, the code had ended up in a state where for each result retrieved from a Solr search, a database query would be made. Not only that, but the query in question was the horribly expensive one I mentioned in point 1.

This will no doubt cause some brow-wrinkling activity if you’re paying attention, as I mentioned that the original intended purpose of the view being used for the expensive query was to push data into Solr – so why, if the data was already in there, would we be accessing it again from the database once we’d pulled it out of Solr?

I can think of a number of good explanations, the best of which might be “My name’s Jon, and I’m a terrible programmer”. The day I get it right 100% of the time is quite probably the day I get to decide which of my Ferrari’s I drive to work, as well as the day that my projects no longer require testers, and I doubt that will ever happen. Maybe I just missed the Ballmer Peak on that day, but whatever happened, I’m just happy when the mistakes I make are as easy to identify and fix as this one was.

Using the SQL Server Database Engine Tuning Advisor

In tandem with this, we also made use of the SQL Server Database Engine Tuning Advisor. This is the next stage of evolution for the Index Tuning Wizard that (I believe) first appeared in SQL Server 2000. The MSDN page for it is here; the short (aka .NET developer) version is that you put a trace file into one end, and out of the other comes a variety of changes you can make to your database to make it go faster.

In order to generate the input, you use SQL Server Profiler with the “Tuning” template. Once this is running, and saving the trace to a file, you need to generate a “typical” load against your application. There are various ways to do this, ranging from fully manual to fully automated. We were fortunate on our project that we had a complete set of Selenium tests for the site and a way of running them as part of the build. I’m hoping that Tom, the consultant responsible for this, will start blogging about it at some point as it’s really interesting stuff. It meant that to generate our trace, all we had to do was start the profiler, kick of the test suite and go for a beer. The next morning, we pushed the resultant file through the tuning advisor and received our recommendations out of the other end. There was little we disagreed with, and most of the recommendations were subsequently applied to the database.

The rest?

You may have noticed that the main focus in this post has been around the way the application accessed the data, rather than the way the database was built. Over the duration of the project, and in addition to the fantastic data architect we had in the team, we’ve had guest appearances by JRJ, Jamie and Simon, so it shouldn’t come as a massive surprise that the database works well. EMC Consulting is rightly proud of the database guys we have on the team and whilst I’m sure that there are many further tweaks that could be made to our databases to add further go-faster stripes, they aren’t needed at the moment.  Optimisation is one of many things that doesn’t benefit by having the word “premature” put in front of it – it’s basically another form of  YAGNI. So, until testing proved otherwise, we were happy to stop at this point and move on to something else.

In the next exciting epsiode of “Optimising an ASP.NET MVC web site”, we look at a pattern for applying application layer caching. Don’t touch that dial!

Please let me know what you think of my blog posts, either by commenting, dropping me an email, or via Twitter.

@jon_george1

One of the things I’ve been involved in over the past couple of months is performance tuning work for my current project (now live at www.fancydressoutfitters.co.uk). One of my EMC Consulting colleagues, Marcin Kaluza, has recently started posting on this subject and I’ve been encouraged by Howard to post some “war stories” of the kind of things I’ve encountered whilst doing this on projects, starting with the most recent.

So first, some background. It’s an public facing website project, and is based on Billy McCafferty’s excellent S#arp Architecture – which means it’s ASP.NET MVC with NHibernate talking to SQL Server 2008 databases. We’re using the Spark View Engine instead of the out of the box one, and the site uses N2 CMS to provide content management capabilities (James posted a while back on the reasons for choosing N2.) Finally, we use Solr to provide our search functionality, integrated using Solrnet. I joined the team a few months into the project, by which point they had laid a firm foundation and were about to be abandoned for 6 weeks by their technical lead who had inconsiderately booked his wedding and an extended honeymoon right in the middle of the project.

When the project was set up it was done so in strictly in accordance with agile principles. A small team was given a fixed date for go-live and the directive to spend the client’s money as if it were their own. One of the first things that happened was the adoption of a number of principles from the excellent 37signals e-book “Getting  Real”. A product backlog was assembled, and then – in accordance with the “build less” maxim – divided into “core” and “non-core” user stories. The core stories were what was essential for go live – things the client couldn’t live without, such as basic search and content management. The non-core stories are things that might enhance the site but aren’t essential – for example, advanced search features such as faceted navigation.

The absolute focus the team maintained on the core functionality and target delivery date has made this one of the best and most successful agile projects I’ve worked on – we reached our go live date on budget and were able to substantially over deliver on functionality. Whilst the site is relatively basic compared to some I’ve worked on, it stands out amongst its peers and provides a great platform for new functionality to be built on.

However, now I’ve extolled the virtues of the approach that was taken, I should talk about the performance optimisation and testing work we did. Since I have some experience from previous projects, I took on the task of testing the site to make sure it could handle an acceptable level of load without falling over in an embarrassing heap. However, before I started on that, we did some optimisation work on the site.

The aim was to hit the major pain points, since we knew performance had degraded over the previous few sprints. Once this was done, we could run some load testing and perform further tuning and optimisation work as required. I was originally intending to write a single post covering the optimisation process, then follow that up with one about the load testing process. However, that resulted in a rather lengthy post, so I’m splitting it up into several parts that I will post over the next week or two:

• Part 1: Introduction (this post)
• Part 2: Database and NHibernate-based repositories
• Part 3: Application caching
• Part 4: View optimisation and output caching
• Part 5: Putting your money where your mouth is 

In addition, I’ve already covered the work we did to correctly configure IIS in my post How to improve your YSlow score under IIS7.

I hope you find these posts interesting – please let me know what you think by leaving a comment.

@jon_george1

On my last two projects, I’ve been involved in various bits of performance tweaking and testing. One of the common tools for checking that your web pages are optimised for delivery to the user is YSlow, which rates your pages against the Yahoo! best practices guide.

Since I’ve now done this twice, I thought it would be useful to document what I know about the subject. The examples I’ve given are from my current project, which is built using ASP.NET MVC, the Spark View Engine and hosted on IIS7.

Make Fewer Http Requests

There are two sides to this. Firstly and most obviously, the lighter your pages the faster the user can download them. This is something that has to be addressed at an early stage in the design process – you need to ensure you balance up the need for a compelling and rich user experience against the requirements for fast download and page render times. Having well defined non-functional  requirements up front, and ensuring that page designs are technically reviewed with those SLAs in mind is an important part of this. As I have learned from painful experience, having a conversation about slow page download times after you’ve implemented the fantastic designs that the client loved is not nice.

The second part of this is that all but the sparsest of pages can still be optimised by reducing the number of individual files downloaded. You can do this by combining your CSS, Javascript and image files into single files. CSS and JS combining and minification is pretty straightforward these days, especially with things like the Yahoo! UI Library: YUI Compressor for .NET. On my current project, we use the MSBuild task from that library to combine, minify and obfuscate JS and CSS as part of our release builds. To control which files are referenced in our views, we add the list of files to our base ViewModel class (from which all the other page ViewModels inherit) using conditional compilation statements.

Images are more tricky, and we’ve made the decision to not bother using CSS Sprites since most of the ways to do it are manual. If this is important to you, see the end of this post for a tool that can do it on the fly.

Use a Content Delivery Network

This isn’t something you can’t really address through configuration. In case you’re not aware, the idea is to farm off the hosting of static files onto a network of geographically distributed servers, and ensure that each user receives that content from the closest server to them. There are a number of CDN providers around – Akamai, Amazon Cloud Front and Highwinds, to name but a few.

The biggest problem I’ve encountered with the use of CDNs is for sites which have secure areas. If you want to avoid annoying browser popups telling you that your secure page contains insecure content, you need to ensure that your CDN has both a http and https URLs, and that you have a good way of switching between them as needed. This is a real pain for things like background images in CSS and I haven’t found a good solution for it yet.

On the plus side I was pleased to see that Spark provides support for CDNs in the form of it’s <resources> section, which allows you to define a path to match against an a full URL to substitute – so for example, you could tell it to map all files referenced in the path “~/content/css/” to “http://yourcdnprovider.com/youraccount/allstyles/css/”. Pretty cool – if Louis could extend that to address the secure/insecure path issue, it would be perfect.

Add an Expires or Cache-Control Header

The best practice recommendation is to set far future expiry headers on all static content (JS/CSS/images/etc). This means that once the files are pushed to production you can never change them, because users may have already downloaded and cached an old version. Instead, you change the content by creating new files and referencing those instead (read more here.) This is fine for most of our resources, but sometimes you will have imagery which has to follow a set naming convention – this is the case for my current project, where the product imagery is named in a specific way. In this scenario, you just have to make sure that the different types of images are located in separate paths so you can configure them independently, and then choose an appropriate expiration policy based on how often you think the files will change.

To set the headers, all you need is a web.config in the relevant folder.  You can either create this manually, or using the IIS Manager, which will create the file for you. To do it using IIS Manager, select the folder you need to set the headers for, choose the “HTTP Response Headers” option and then click the “Set Common Headers” option in the right hand menu. I personally favour including the web.config in my project and in source control as then it gets deployed with the rest of the project (a great improvement over IIS6).

Here’s the one we use for the Content folder.

This has the effect of setting the HTTP Expires header to the date specified.

And here’s what we use for the product images folder.

This writes max-age=604800 (7 days worth of seconds) into the Cache-Control header in the response. The browser will use this in conjunction with the Date header to determine whether the cached file is still valid.

Our HTML pages all have max-age=0 in the cache-control header, and have the expires header set to the same as the Date and Last-Modified headers, as we don’t want these cached – it’s an ecommerce website and we have user-specific infomation (e.g. basket summary, recently viewed items) on each page.

Compress components with Gzip

IIS7 does this for you out of the box but it’s not always 100% clear how to configure it, so, here’s the details.

Static compression is for files that are identical every time they are requested, e.g. Javascript and CSS. Dynamic compression is for files that differ per request, like the HTML pages generated by our app. The content types that actually fall under each heading are controlled in the IIS7 metabase, which resides in the file C:\Windows\System32\inetsrv\config\applicationHost.config. In that file, assuming you have both static and dynamic compression features enabled, you’ll be able to find the following section:

As you can see, configuration is done by response MIME type, making it pretty straightforward to set up…

One gotcha there is that IIS7 is configured out of the box to return a content type of application/x-javascript for .JS files, which results in them coming under the configuration for dynamic compression.This is not great – dynamic compression is intended for frequently changing content so dynamically compressed files are not cached for future requests. Our JS files don’t change outside of deployments, so we really need them to be in the static compression category.

There’s a fair bit of discussion online as to what the correct MIME type for Javascript should be, with the three choices being:

• text/javascript
• application/x-javascript
• application/javascript

So you have two choices – you can change your config, either in web.config or applicationHost.config to map the .js extension to text/javascript or application/javascript, which are both already registered for static compression, or you can move application/x-javascript into the static configuration section in your web.config or applicationHost.config. I went with the latter option and modified applicationHost.config. I look forward to a comment from anyone who can explain whether that was the correct thing to do or not.

Finally, in the <system.webserver> element of your app’s main web.config, add this:

Once you’ve got this configured, you’ll probably open up Firebug or Fiddler, hit your app, and wonder why your static files don’t have the Content-Encoding: gzip header. The reason is that IIS7 is being clever. If you return to applicationHost.config, one thing you won’t see is this:

You’ll just have an empty <serverRuntime /> tag. The values I’ve shown above are the defaults, and the interesting ones are “frequentHitThreshold” and “frequentHitTimePeriod”. Essentially, a file is only a candidate for static compression if it’s a frequently hit file, and those two attributes control the definition of “frequently hit”. So first time round, you won’t see the Content-Encoding: gzip header, but if you’re quick with the refresh button, it should appear. If you spend some time Googling, you’ll find some discussion around setting frequentHitThreshold to 1 – some say it’s a bad idea, some say do it.  Personally, I decided to trust the people who built IIS7 since in all likelihood they have larger brains than me, and move on.

There are a couple of other interesting configuration values in this area – the MSDN page covers them in detail.

Minify Javascript and CSS

The MSBuild task I mentioned under “Make fewer HTTP requests” minifies the CSS and JS as well as combining the files.

Configure Entity tags (etags)

ETags are a bit of a pain, with a lot of confusion and discussion around them. If anyone has the full picture, I’d love to hear it but my current understanding is that when running IIS, the generated etags vary by server, and will also change if the app restarts. Since this effectively renders them useless, we’re removing them. Howard has already blogged this technique for removing unwanted response headers in IIS7. Some people have had luck with adding a new, blank response header called ETag in their web.config file, but that didn’t work for me.

Split Components Across Domains

The HTTP1.1 protocol states that browsers should not make more than two simultaneous connections to the same domain. Whilst the most recent browser versions (IE8, Chrome and Firefox 3+) ignore this, older browsers will normally still be tied to that figure. By creating separate subdomains for your images, JS and CSS you can get these browsers to download more of your content in parallel. It’s worth noting that using a CDN also helps here. Doing this will also help with the “Use Cookie Free Domains For Components” rule, since any domain cookies you use won’t be sent for requests to the subdomains/CDN. However, you will probably fall foul of the issue I mentioned earlier around mixing secure and insecure content on a single page.

A great tool for visualising how your browser is downloading content is Microsoft Visual Roundtrip Analyzer. Fire it up and request a page, and you will be bombarded with all sorts of information (too much to cover here) about how well the site performs for that request.

Optimise Images

Sounds obvious, doesn’t it? There are some good tools out there to help with this, such as Smush.it, now part of YSlow. If you have an image heavy website, you can make a massive difference to overall page weight by doing this.

Make favicon.ico Small and Cacheable

If you don’t have a favicon.ico, you’re missing a trick – not least because the browser is going to ask for it anyway, so you’re better returning  one than returning a 404 response. The recommendation is that it’s kept under 1Kb and that it has a fairly long expiration period.

WAX – Best practice by default

On my last project, I used the Aptimize Website Accelerator to handle the majority of the things I’ve mentioned above (my current project has a tight budget, so it’s not part of the solution for v1). It’s a fantastic tool and I’d strongly recommend anyone wishing to speed their site up gives it a try. I was interested to learn recently that Microsoft have started running it on sharepoint.microsoft.com, and are seeing a 40-60% reduction in page load times. It handles combination and minification of CSS, Javascript and images, as well as a number of other cool tricks such as inlining images into CSS files where the requesting browser supports it, setting expiration dates appropriately, and so on.

Update: Microsoft’s SharePoint team have blogged about using WAX on sharepoint.microsoft.com, with the detail being provided by Ed Robinson, the CEO of Aptimize.

@jon_george1