Into the core - Independent Sitecore developer viewpoints

This blog has moved

2010-06-03T09:14:00.001+01:00

This blog is now located at http://intothecore.cassidy.dk/. You will be automatically redirected in 30 seconds or you may click here. For feed subscribers, please update your feed subscriptions to http://intothecore.cassidy.dk/feeds/posts/default.

Partial Cache Clearing in Sitecore

2009-09-03T17:57:00.001+01:00

For a very long time, some very serious performance issues have affected certain types of Sitecore deployments. It has to do with scaling a Sitecore solution into webfarms (multiple content delivery servers) and the use of the Sitecore Staging Module.

In very short summary, if the website is updated frequently (say, for instance, an editorial staff of 10 or so, posting relevant market updates) AND the website is under heavy load, most practical uses of the tool Sitecore recommends and supports will more or less kill performance on your SQL Servers or whatever storage mechanism you have in place.

I’ve never blogged in detail about this issue, as I didn’t have a client myself that was affected directly by this issue. Paul George has blogged about it in detail however, and if you want to learn more about what this issue is and if it could be affecting you, take a look at these posts:

So why write about it now?

The good news

Alex Shyba posted about a new Sitecore Shared Source module that was just made publically available; SitecoreStager – Sitecore Partial (item) Cache Clearing Module.

In short, instead of just uncritically clearing the entire cache on your target server, dropping user sessions, putting out the cat and so on – the SitecoreStager will instead execute a partial cache clear and in essence just clear items from the cache that were affected by the publishing operation.

This is very good news indeed, and for those clients who have been affected by this problem I am sure it’s tipped hats all ‘round :-)

The bad news

This is actually something that has been nagging me a bit for some time now, and has just been refreshed by this release.

It is Shared Source.

Don’t get me wrong. I think the Sitecore Shared Source Library is an excellent idea. I have code in there myself, and it’s a perfect place to go look for those “there has to be someone else who had this problem and solved it” code snippets and field types and whatever it may be.

But it is also, for obvious reasons, unsupported by Sitecore themselves. I mean, how could they? Half the modules and code pieces in there come from independent sources like myself – and it wouldn’t be reasonable to expect Sitecore to support them.

But what about the code Sitecore release to the library themselves?

Would you not agree; “With Sitecore, you can have a team of editors publishing content to an enterprise level site, and performance will still rock” is a statement you’ll find (albeit probably not verbatim) in Sitecore marketing material? But it is unsupported?

I feel somewhat the same for the Multiple Sites Manager… shouldn’t a standard Sitecore have a solution for regular (advanced) editors to set up new websites without having to edit configuration files? Or is that limited to Foundry licenses only?

What I mean is

Sitecore marketing materials doesn’t exactly help anyone much, when they boast “Comes with Blog Modules, Wiki Modules and even Microsoft Dynamix and Microsoft BizTalk integration (yes…)”; yet offers little or no actual support other than pointing the users, the licensees, the Sitecore implementors (like myself) in the direction of the Shared Souce Library, shrug and go… “From here, you’re on your own”.

Kudos for making this module. I think this belongs in the core package however, fully supported and updated with every Sitecore release. That’s my 2 cents worth.

Code Monkeys?

2009-06-23T19:04:00.001+01:00

Before I begin, I’m breaking routine here – this post is in no way Sitecore related.

So here I was, reading through my blogroll, catching up on bits and bobs from around the world. And then one of the sites decides to confront me with the following ad:

I am, by own admission, quite possibly a bit over sensitive to the idea that developers are interchangeable code monkeys. I’ve worked in the internet industry pretty much since the inception in the mid 90s, and I will never subscribe to the notion that one developer (me) can easily be replaced by a developer working offshore for as low as $ USD 7.00 a day.

But then again, maybe that’s just me ;-)

Am I missing a point here?

Just because you can doesn’t mean you should

2009-06-05T13:38:00.001+01:00

For those of you who don’t know this; I make my living as a Sitecore Professional Services Consultant. Understand this in the context of working with the Sitecore product as a consultant, I don’t actually work for Sitecore the company.

My job, if you will, consists primarily of establishing contact with newly started Sitecore Partners or want-to-become Partners, and bringing to them whatever skills I can offer to help them take those first shaky steps when bringing their first Sitecore Project to life.

Don’t worry, am not going to start any kind of sales pitch here, that’s not what ~~blogs~~ this blog ~~are~~ is for. I’m only telling, so you know what sort of context this post is in and where my experiences are coming from.

As you can maybe imagine (or even remember still?); the myriad of questions in relation to Sitecore coming from developers, sales people, project managers and so on are many and not far between. Nothing wrong with that, obviously, we all have to learn. Mostly the questions start out around the capabilities of the product, “can” questions. After this, they move into “how”, and ultimately (in the cases where I’m actually lucky enough to be around for the entire project) “when”.

"Can” questions

Initially, when requirements are being gathered, it’s a heap of “Can Sitecore do this?” type of questions. To name just a very few examples:

Can you integrate Sitecore with SharePoint?
Can you use the AJAX toolkit on a Sitecore site?
Can Sitecore deliver Flash content?
Can we build a database of our people and offices in Sitecore and have them shown on the site?
Can we have search options on our site?
Can Sitecore integrate with Google Analytics?
Can we implement breadcrumbs with Sitecore?

There are, of course, many more questions that are usually asked – as I’m sure any member of the Sitecore sales team will tell you as well. Incidentally, the answer to all of the above questions is “yes”.

And now we’re getting to the point. Working with Sitecore – how often is it, that you actually have to sat “No”. “Sorry. Can’t do that with Sitecore”? While I have no statistics on it, I can still state without shaking my hand that this very rarely happens. Sitecore allows you to say “Yes. Can do” to almost any requirement, no matter how far fetched it may seem or even be.

Let’s not get too carried away, however, this is not quite as amazing as it might appear on the surface. If we boil it down to the bone, Sitecore can be described as an “ASP.NET application that adds Content Management services to your ASP.NET websites”. Right – I’m sure there’s a hundred different marketing spins to be made here, but bear with me… ;-)

Sitecore is an ASP.NET application. It sits, talks, walks and sounds like an ASP.NET application. And here’s the good bit – and the reason you can answer “yes” to almost any requirement – anything you can do in ASP.NET, you can continue doing – Sitecore or no Sitecore involved.

Even the Sitecore product itself is flexible to the bone - (almost) everything is based on configuration files, and they can be tweaked and twisted or completely rewritten – in case there’s a particular feature you are missing, or if there’s a certain way Sitecore works that you want to change or even remove entirely. A blessing, right?

Wrong! And on after-thought; “Wrong… mostly!”

Don’t get me wrong. I totally get where Sitecore is coming from, in wanting to create as flexible a platform as possible. As any software vendor on the market; the more requirements you can say “yes” to, the more sales you are likely to get. This is as simple and obvious as can be, really.

I’m just saying; maybe “Yes. But…” is a better answer. Let’s take a look at some more “can” questions. These are not fictional by the way, they are real questions asked by real people.

“Can” questions part 2

Can you rewrite how Sitecore creates new items, so that all language versions are created instead of just the current language you are editing?
Can you use ASP.NET Master Pages and Content Pages with Sitecore? Does Sitecore support nested Master Pages?
Can you work with Web Parts in Sitecore?
- This one deserves a closer examination; and after doing that what is usually really asked here is, “Can I put my page into edit mode, and add/remove Web Parts on different placeholder areas of the page?”
Can I use Microsoft Word to edit my pages?
Can I use the Microsoft MVC Framework with Sitecore?

Do you see where this is going?

What are these people really asking for? Features?

Nope. In the vast majority of cases, these are not feature requests. These people are looking for familiarity.

Yep. That’s right. Familiar ground to stand on. Something known, as opposed to something unknown. Can we blame them? Absolutely not. I think we all have a bit of fear of the unknown, to one degree or another.

And of course, Sitecore is fully aware of this as well. They have released features specifically to address some of these very questions already, and there are more to come. That way, we can continue saying “yes” and everyone wins. Just keep this point in mind however – familiarity. Not features. More often than not.

The solution

I guess the word “solution” is not really appropriate, as it seems to indicate there was a problem to begin with. Look behind the question. Find out why it is being asked in the first place, and then work from there.

Why does the person want to change how Sitecore creates language versions?

Turns out, this is apparently how other major CMS vendors do it. Now I happen to think Sitecore is doing this the right way, but this person came from a different background with different experience. He was looking for a way to solve (amongst other things) content translation flows – and was maybe not aware of the various tools and gadgets that Sitecore provide for this purpose.

Master Pages?

Think “overworked .NET developer who really cannot be bothered to try and figure out how Sitecore constructs the pages it delivers”.

I certainly get where he or she is coming from. But don’t go chasing down this road – sit down and show (don’t tell; show) how a Sitecore “Layout” and an ASP.NET “Master Page” is more or less essentially the same thing. (I know… but really – in most cases, this is just semantics). So “placeholders”, not “content placeholders”. “Layouts”, not “Master Pages”.

Am sure you’re seeing the pattern here, by now :-)

All I’m saying is; “Yes. You absolutely CAN reconfigure Sitecore, so that security is completely disabled, users and roles come off a scan of your table napkin, your coffee machine starts automatically when you publish AND it will even offer background music when content editing”. Ok I’m being ironic, obviously, but see the point is this.

Just because you can doesn’t mean you should

Don’t go mucking about with Sitecore, jumping through hoops to make it act a certain way. Chances are – and most times – you aren’t dealing with a real requirement at all. All too often have I arrived at a “young” Sitecore shop and seen a development team dive right into a complete reconfiguration of how Sitecore works. Pipelines altered and skewed and tonnes of bespoke code developed because “Otherwise we couldn’t meet our clients requirements”. Err… WHAT requirements were those exactly?

I mean, come on. I’ve done lots and lots of Sitecore sites now – and I can count the number of times I’ve actually had to modify standard Sitecore behaviour on maybe one hand. This does not include adding new modules or field types, or anything like that – I’m talking about core changes such as altering the publishing pipeline, messing about with security resolvers and so on.

Not pointing fingers at anyone in particular here. If I were, I would have to point to myself first – I’m as guilty of trying to change something from what it is into something that is familiar as I gather anyone else is.

I just think it’s something we should all keep in mind.

And oh… before I end.

Sometimes the question DOES lead to a “legitimate” feature request ;-)

Listing “Related Articles” with Sitecore using the LinkDatabase

2009-05-15T00:37:00.001+01:00

Seems I am on a writing streak this week. Am taking a week off, you see, from my normal everyday Sitecore Consulting, and seem to have a bit of time on my hands to catch up on some of all the posts I’ve been meaning to write for a while. Don’t worry; after this I will probably be way too busy again for a while to find time to post ;-)

So I catching up on StackOverflow the other day, and an interesting question was posed; “How to find related items by tags in Lucene.NET”.

And while there probably IS a way to actually do this with Lucene.NET; I remember my initial thought was “but why go through all the hassle of configuring and setting it up to do this?”. Not only would it matter things from an Operations point of view; it would require more code and more code that was completely dependant on specific configuration settings in the Lucene indexes.

Now, let me be very clear, I am no big expert on Lucene. There are many of you out there who know it well, and would probably be able to cook up a solution to answer the guys question using it. As for myself, I try to keep as much arcane configuration out of any project I am involved in – especially to solve a problem such as this, where Sitecore pretty much gives you the tools you need to solve it straight out of the box.

So anyway. Guy was asking in a Lucene context, but was looking for proposals. And I decided to give it a whirl, mocked up some pseudo code to solve the problem, and that was that. But see; everyone can write pseudo-code :P And it’s only fair I put my err… code where my mouth is, and write up a real example of how this can be achieved in a manner I explained. Here goes.

Setting it up in Sitecore

I start by making up two templates:

1) “Simple Value”, which will be used to organise the meta tags I will be drawing upon.

It has no fields.

2) “Article”, which I will use to demonstrate how to implement “Related Articles” functionality.

I then set up a meta-structure that I will be using to tag up my articles, and ultimately draw out related articles. I don’t fill out the entire structure, nor do I mean to imply this structure is perfect. But it is enough to demonstrate the point, and should be easy enough to follow. All the tags are based on the “Simple Value” template.

After this, I go through the somewhat tedious task of setting up a number of articles that are tagged in different ways.

For now, I type and tag in 7 articles; like this:

Name: Ben Hur

Tags: O2 Arena, Theatre

Name: Britney Spears

Tags: O2 Arena, Pop, Concert

Name: Depeche Mode

Tags: O2 Arena, Alternative, Concert

Name: Michael Jackson

Tags: O2 Arena, Pop, Concert

Name: Nickelback

Tags: O2 Arena, Rock, Concert

Name: Pet Shop Boys

Tags: O2 Arena, Pop, Concert

Name: War of the Worlds

Tags: O2 Arena, Theatre

I should probably go on for a while longer if I really wanted to go all-out in demonstrating this. However, I do have enough now, and it’ll have to do. I hate typing in test data ;-)

Before I go on, I should explain exactly how I intend to deduce what “related articles” should be. It can be done and determined in many ways – but I am proceeding exactly in the manner that was originally in question on StackOverflow. The rule can be described as two statements:

1) An article is related if it shares one or more tags with the source article

2) The more tags it shares, the more relevant it becomes (i.e. should appear higher on the list)

Lastly, I set up a blank .ASPX page in my webroot named “TestRelated.aspx”, and I quickly mock up two DomainObjects that I will build upon for this functionality.

SimpleValue.cs

using CorePoint.DomainObjects.SC;
using CorePoint.DomainObjects;

namespace Website.Related
{
    [Template("user defined/simple value")]
    public class SimpleValue : StandardTemplate
    {
    }
}

Article.cs

using System;
using System.Collections.Generic;
using CorePoint.DomainObjects.SC;
using CorePoint.DomainObjects;

namespace Website.Related
{
    [Template("user defined/article")]
    public class Article : StandardTemplate
    {
        [Field("title")]
        public string Title { get; set; }

        [Field("text")]
        public string Text { get; set; }

        [Field("tags")]
        public List<Guid> Tags { get; set; }
    }
}

And finally, in my TestRelated.aspx.cs, I add a bit of code to test that everything is as expected.

public partial class TestRelated : System.Web.UI.Page
{
    protected void Page_Load( object sender, EventArgs e )
    {
        var director = new SCDirector();

        List<Article> articles = director.GetChildObjects<Article>( "/sitecore/content/global/articles" );
        foreach ( Article article in articles )
        {
            // Get the SimpleValues (name) from the tag Guids
            var simpleValues = article.Tags.ConvertAll<string>( a => 
                        { 
                            return director.GetObjectByIdentifier<SimpleValue>( a ).Name; 
                        } );
            StringBuilder sb = new StringBuilder();
            simpleValues.ForEach( sv => sb.Append( sv + ' ' ) );

            Response.Write( string.Format(
                             "Name: {0}<br />Tags: {1}<br /><br />",
                             article.Name,
                             sb.ToString() ) );
        }
    }
}

So far so good. I run the code, and I get a replica of the list I already showed:

Name: Ben Hur
Tags: O2 Arena Theater
Name: Britney Spears
Tags: Pop Concert O2 Arena
Name: Depeche Mode
Tags: O2 Arena Concert Alternative
Name: Michael Jackson
Tags: Pop Concert O2 Arena
Name: Nickelback
Tags: Rock Concert O2 Arena
Name: Pet Shop Boys
Tags: O2 Arena Concert Pop
Name: War of the Worlds
Tags: O2 Arena Musical

Excellent. After all this, I am now ready to proceed to the good stuff ;-)

Finding Related Articles using the Sitecore LinkDatabase

Having an Article entity in place, makes this an obvious place to add functionality such as Related Articles. I could either add it as a Lazy Load property named “Related Articles”, or I could write a method named “GetRelatedArticles()”. This is mostly down to aesthetics and practices; personally I prefer the first option.

I expand the Article.cs with a little bit of code. The original pseudo-code I suggested, is entered in comments, for reference.

private int _referenceCount;
List<Article> _RelatedArticles = null;
public List<Article> RelatedArticles
{
    get
    {
        if ( _RelatedArticles == null )
        {
            _RelatedArticles = new List<Article>();
            var referenceCount = new Dictionary<Guid, int>();

            // for each ID in tags
            foreach ( Guid id in Tags )
            {
                var sv = Director.GetObjectByIdentifier<SimpleValue>( id );

                // Personal note: In this particular instance, performance
                // could be gained here, but not loading up full articles
                // via DomainObjects but hitting the LinkDatabase directly instead

                // get all documents referencing this tag
                List<Article> articles = sv.GetReferrers<Article>();

                // for each document found
                articles.ForEach( a =>
                    {
                        if ( a.Id != Id )
                        {
                            // if master-list contains document; 
                            if ( referenceCount.ContainsKey( a.Id ) )
                                referenceCount[ a.Id ]++; // increase usage-count
                            else // else; 
                                // add document to master list
                                referenceCount[ a.Id ] = 1;
                        }
                    } );
            }

            // Now we have a list of all the relevant guids being referenced on all tags
            // on this article. Load them up, and stamp them with the reference count
            foreach ( var key in referenceCount.Keys )
            {
                var relatedArticle = Director.GetObjectByIdentifier<Article>( key );
                relatedArticle._referenceCount = referenceCount[ key ];
                _RelatedArticles.Add( relatedArticle );
            }
            
            // sort master-list by usage-count descending
            _RelatedArticles.Sort( ( a, b ) => b._referenceCount.CompareTo( a._referenceCount ) );
        }

        return _RelatedArticles;
    }
}

And to test if what I’m getting from this is what I expect, I also add some code to my TestRelated.aspx so it becomes:

protected void Page_Load( object sender, EventArgs e )
{
    var director = new SCDirector();

    List<Article> articles = director.GetChildObjects<Article>( "/sitecore/content/global/articles" );
    foreach ( Article article in articles )
    {
        // Get the SimpleValues (name) from the tag Guids
        var simpleValues = article.Tags.ConvertAll<string>( a => 
                    { 
                        return director.GetObjectByIdentifier<SimpleValue>( a ).Name; 
                    } );
        StringBuilder sb = new StringBuilder();
        simpleValues.ForEach( sv => sb.Append( sv + ", " ) );

        Response.Write( string.Format(
                         "Name: {0}<br />Tags: {1}<br />Related Articles: ",
                         article.Name,
                         sb.ToString() ) );

        article.RelatedArticles.ForEach( ra =>
                Response.Write( string.Format( "{0},", ra.Name ) ) );

        Response.Write( "<hr />" );
    }
}

And after all this, I am pleased to find a result looking like:

Name: Ben Hur
Tags: O2 Arena, Theater,
Related Articles: Michael Jackson,Britney Spears,Depeche Mode,Nickelback,Pet Shop Boys,War of the Worlds,

Name: Britney Spears
Tags: Pop, Concert, O2 Arena,
Related Articles: Michael Jackson,Pet Shop Boys,Depeche Mode,Nickelback,Ben Hur,War of the Worlds,

Name: Depeche Mode
Tags: O2 Arena, Concert, Alternative,
Related Articles: Britney Spears,Michael Jackson,Nickelback,Pet Shop Boys,War of the Worlds,Ben Hur,

Name: Michael Jackson
Tags: Pop, Concert, O2 Arena,
Related Articles: Britney Spears,Pet Shop Boys,Depeche Mode,Nickelback,Ben Hur,War of the Worlds,

Name: Nickelback
Tags: Rock, Concert, O2 Arena,
Related Articles: Britney Spears,Depeche Mode,Pet Shop Boys,Michael Jackson,Ben Hur,War of the Worlds,

Name: Pet Shop Boys
Tags: O2 Arena, Concert, Pop,
Related Articles: Britney Spears,Michael Jackson,Depeche Mode,Nickelback,War of the Worlds,Ben Hur,

Name: War of the Worlds
Tags: O2 Arena, Musical,
Related Articles: Ben Hur,Britney Spears,Depeche Mode,Nickelback,Pet Shop Boys,Michael Jackson,

The first thing that strikes me is; my meta data and test data probably aren’t extensive enough to really see this functionality in full effect. They all look almost the same.

However, I can determine that it works as expected. “Britney Spears”, “Michael Jackson” and “Pet Shop Boys” all share the same 3 meta tags. They SHOULD in all instances suggest the “one left out” on top of the list as “Related Articles”. And they all do; I’ve marked them in bold and underline. Also note that the “Depeche Mode” concert in O2 Arena lists other concerts (although of different music genre) before it proceeds to list the musicals and theatre plays.

It works :-)

A few notes on performance

In this post, I’ve deliberately not focused excessively on performance implications. Don’t worry – it’s not at all bad. But in “real life”; there are still obvious places in this code where you could potentially gain a significant amount of performance. As everyone will know; I/O operations are by an order of magnitude some of the most expensive calls we can make, and there is definitely a few places you could set in here.

A few suggestions I would look into if I were to take this code live:

Code up a TagController; that will eventually act as a cache for all the tags in your solution. Load up the tags only once, and don’t repeatedly re-load them in your loops.
In this case, bypass the very convenient .GetReferrers() method provided by DomainObjects and go through the extra work of working with the LinkDatabase directly yourself. For this part of the algorithm (counting up how many times a given ID is referencing your tag), you don’t really need to load up the Sitecore Item – something .GetReferrers() will automatically do. I will put this on the TODO list for DomainObjects.
And – as ALWAYS – don’t forget to configure caching for whatever sublayouts and/or user controls you are calling this functionality on.

That’s it for this time. I hope you found this useful :-)

Working with web.config include files in Sitecore 6

2009-05-13T01:04:00.001+01:00

In my previous post about Working with multiple content databases; Lars Floe Nielsen made a comment about something I’ve been meaning to write about for a long time.

Configuration files. Such a pain, aren’t they?

Anyone who has ever stepped through 6 Sitecore upgrades and meticulously stepped through the web.config change instructions line by line will know what I mean. Would be so much easier to just replace your web.config with the one matching the latest Sitecore version you were upgrading to.

Or what about your environments? Dev environment, Staging environment, Live environment, Slave server environment? All with different configuration settings. This has already been blogged about, and I am not going to dig particularly deep into this topic in this post.

Starting from Sitecore 6 (actually, V5, but I’ve had a very hard time tracking more information down on it than can be found in Alexeys post on the matter), Sitecore actually introduced a really neat new functionality. It’s called “Web Config Patching”, but to be honest I don’t personally like the term “patching” being used in this context, even if this IS technically what the functionality does.

So far, I have not really been able to locate much in terms of official documentation on this subject (searching SDN directly provides very few clues), so most of my knowledge on it comes from personal experience, chatting with other Sitecore consultants/investigators, studying other configuration include files and spiced with generous dosages of Reflector.

In the “What’s new” released for Sitecore 6, the functionality gets the following mention:

“Previous versions of Sitecore CMS forced administrators to make direct changes to configuration settings in the web.config file manually. This led to challenges locating local configuration changes as opposed to modifications made by Sitecore when upgrading to a new version of Sitecore. Sitecore 6 offers a smart solution: web.config modifications can now be made in a separate XML file, stored under the /App_Config/Include folder, which Sitecore reads in at startup time after loading the web.config file. The folder contains several example files which illustrate how to use this feature. The Sitecore 6 configuration factory reads the include config files”

The information appears out of date however, and no such “example files” can be found in any version of Sitecore 6 I have had my hands on.

Anyway. On we go.

So how and where does it work?

To make good use of config includes, one must first understand how Sitecore implements it. And to get some idea of this, one must know a little bit about how a web.config file is organised.

If you open up a standard Sitecore web.config and look near the top, the first thing you will see will be looking something like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="sitecore" type="Sitecore.Configuration.ConfigReader, Sitecore.Kernel" />
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, 
                                  Sitecore.Logging" />
    <sectionGroup name="system.web.extensions" type="System.Web.Configuration.SystemWebExtensionsSectionGroup, 
                                                     System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                                     PublicKeyToken=31BF3856AD364E35">
      <sectionGroup name="scripting" type="System.Web.Configuration.ScriptingSectionGroup, 
                                           System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                           PublicKeyToken=31BF3856AD364E35">
        <section name="scriptResourceHandler" type="System.Web.Configuration.ScriptingScriptResourceHandlerSection, 
                                                    System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                                    PublicKeyToken=31BF3856AD364E35" 
                 requirePermission="false" allowDefinition="MachineToApplication" />
        <sectionGroup name="webServices" type="System.Web.Configuration.ScriptingWebServicesSectionGroup, 
                                               System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                               PublicKeyToken=31BF3856AD364E35">
          <section name="jsonSerialization" type="System.Web.Configuration.ScriptingJsonSerializationSection, 
                                                  System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                                  PublicKeyToken=31BF3856AD364E35" 
                   requirePermission="false" allowDefinition="Everywhere" />
          <section name="profileService" type="System.Web.Configuration.ScriptingProfileServiceSection, 
                                               System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                               PublicKeyToken=31BF3856AD364E35" 
                   requirePermission="false" allowDefinition="MachineToApplication" />
          <section name="authenticationService" type="System.Web.Configuration.ScriptingAuthenticationServiceSection, 
                                                      System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                                      PublicKeyToken=31BF3856AD364E35" 
                   requirePermission="false" allowDefinition="MachineToApplication" />
          <section name="roleService" type="System.Web.Configuration.ScriptingRoleServiceSection, 
                                            System.Web.Extensions, Version=3.5.0.0, Culture=neutral, 
                                            PublicKeyToken=31BF3856AD364E35" 
                   requirePermission="false" allowDefinition="MachineToApplication" />
        </sectionGroup>
      </sectionGroup>
    </sectionGroup>
  </configSections>

What is declared here, are the different Configuration Sections that ASP.NET can expect to find in the configuration file. Some of them are there to support ASP.NET, and some of them are put in there by Sitecore. You can learn more about the format of ASP.NET configuration files here.

Basically, what this then means is, that various “top level” configuration sections can be expected to appear in the web.config file we are looking at, and ASP.NET will (via the “type” attribute) know how to parse them. For normal every day use, most of us have probably been able to just use <appSettings> for whatever configuration we needed – but for configuring a complex application such as Sitecore, this just won’t be enough. Fortunately this is why ASP.NET allows us to create our own configuration sections with our own configuration handlers; and that is exactly what Sitecore has been doing for a very long time.

Now. Keeping in mind what I wrote above; Sitecore came up with a system that allows the include of configuration files. Tying that into what we just learned; to find and use this functionality we must then look in the config section that Sitecore provides. Not surprisingly, this section is called <sitecore> and this is where you configure the vast majority of what you need to do, to get your Sitecore installation up and running the way you want it.

Config Include only works in this configuration section

First thing to keep in mind, when using this technology.

This means it won’t work for <appSetting> configuration settings. Don’t worry about it – Sitecore has a perfectly good replacement for it; I’ll demonstrate in a bit.

How to set it up?

Here’s a bit of good news. There’s nothing really to set up. Sitecore comes with this functionality enabled out of the box, and all you need to do is to tap into it and use it.

If you open up Windows Explorer and navigate to /Website/App_Config/Include, you will (probably) find an empty folder. This is a directory that Sitecore is actively watching, for any additions or changes to it’s base web.config file. Remember I said how it was not fully correct to call this “config include”? This is because Sitecore actually offers more than just including more configuration files; it also allows you to edit existing configuration data defined in web.config. As long as it sits in the <sitecore> section :-)

As so often before when I am testing something; I create a new .ASPX file (with codebehinds) in the root of my website; I name it “TestInclude.aspx”, and I type the following code into the class Visual Studio generates for me:

public partial class TestInclude : System.Web.UI.Page
{
    protected void Page_Load( object sender, EventArgs e )
    {
        Response.Write( "The value of setting 'TestInclude' is: " + 
                        Sitecore.Configuration.Settings.GetSetting( "TestInclude", "Undefined" ) );
    }
}

At this point, the result I get when running the page is entirely as expected; “The value of setting 'TestInclude' is: Undefined”

Notice how the Sitecore API equivalent is much more elegant than the ASP.NET standard handling which would achieve the above in the <appSettings> section.

string val;
if ( System.Configuration.ConfigurationManager.AppSettings[ "TestInclude" ] != null )
    val = System.Configuration.ConfigurationManager.AppSettings[ "TestInclude" ];
else
    val = "Undefined";

But we’re not there yet. I then proceed to create a “New File” in the folder I mentioned above; /App_Config/Include and name it “TestInclude.config”

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <setting name="TestInclude" value="This value comes from TestInclude.config"/>
    </settings>
  </sitecore>
</configuration>

I run my .ASPX page again; and this time I get the result I was hoping for. “The value of setting 'TestInclude' is: This value comes from TestInclude.config”.

Great! :-) Things are working as expected. And I now have my own configuration files in a nice isolated area that can be easily packaged and deployed WITHOUT needing to worry (much) about the version of Sitecore that may be in place; and without needing to touch the original web.config in any way what so ever.

There’s another benefit; or at least in a majority of cases this is a benefit. Making modifications to your config include files take effect (almost) instantly and do not recycle your application pool.

Updating your config files will not force your website to reset

Another important fact to keep in mind. For better and (sometimes) for worse.

Notice how this is not limited to work with only <settings>. Anything in the Sitecore configuration structure can be added in your include file. If I wanted to add a new XSL helper, for instance, I would expand my file like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <xslExtensions>
      <extension mode="on" type="CorePoint.XslHelpers.XslHelper, CorePoint.Library" 
                 namespace="http://www.corepoint-it.com/library/xslhelper" singleInstance="true" />
    </xslExtensions>

    <settings>
      <setting name="TestInclude" value="This value comes from TestInclude.config"/>
    </settings>
  </sitecore>
</configuration>

One last thing to mention about these include files before proceeding is; you can have as many of them as you like. They need to end in .config, but other than that there are no limitations. You can even create sub folders to your App_Config/Include directory and place your .config files there if you prefer; they too will be picked up by Sitecore’s configuration system.

More advanced work with your config include files

In the example I just went through, I adeptly (or maybe not…) skipped explaining part of the reason the config include file I created looks the way it does. What I did was to work with the include system in it’s simplest form. If you picture in your mind your original web.config file, and then merge my XML on top of it; you have a pretty good idea of what I have just done.

And this is fine; for settings. After all, a setting is a setting, and it matters not exactly WHERE in the configuration file the setting appears.

But what about the times when it does matter? Like for Sitecore pipelines for instance; I can assure you the order of which these pipelines executes is NOT irrelevant.

Positioning your configuration within the web.config is fortunately easily achieved. A few examples probably explain it better than I can type myself out of. So here goes.

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <httpRequestBegin>
        <!-- Insert own pipeline processor as the first element of the pipeline -->
        <processor patch:before="*[1]" type="CorePoint.Tracking.RequestTracker, CorePoint.Library" />

        <!-- Insert own pipeline processor right after the Language Resolver -->
        <processor patch:after="*[@type='Sitecore.Pipelines.HttpRequest.LanguageResolver, Sitecore.Kernel']"
                   type="CorePoint.Tracking.LanguageTracker, CorePoint.Library" />
      </httpRequestBegin>
    </pipelines>
    
    <settings>
      <setting name="TestInclude" value="This value comes from TestInclude.config"/>
    </settings>
  </sitecore>
</configuration>

As you can probably see, fairly advanced stuff can be done with configuration files. Most of this syntax and form I have exclusively from Reflector use, and I may not have it spot on correct. Finding official documentation on this topic has proven to be next to impossible. Except for lots of references on various comments around the web (recommended practise is to use config includes or “auto-includes” as they are also called) of course – but knowing HOW to use them is what this post is all about :-)

I would love to know how one can:

Remove an existing configuration entirely
Replace an existing functionality entirely

Both seem possible from digging around in Reflector – but given that this is actually a fairly involved process to test (at 2am in the morning), I chose to let the matter rest for this time. I will get back with an update if and when I learn more.

In summary

Configuration include files is probably one of the features I personally like very much from an operational perspective in Sitecore 6
The functionality is way under-documented; but hopefully now this post can help you get started

So please; no more 3-page documents describing how to “merge” your configuration into web.config for <insert your module/functionality name here> :P

You can modify config include files without resetting your website AppPool
And lastly, it only works in the <sitecore> configuration section. Don’t attempt it for <system.web> or <system.webserver> for instance, it won’t work.

Working with multiple content databases in Sitecore 6

2009-05-08T17:41:00.001+01:00

One of the very neat things about Sitecore, is the way the architecture allows you to mould, shape, and work with the configuration files to come up with an implementation that suits your purpose.

As the title of this post will suggest, I will be taking a look at Sitecore databases in this post; and how you are free to work with as many of them as you see fit in your projects.

For sake of argument, let’s say that you were tasked with expanding an existing Sitecore website with a Products database. Potentially, this database would be holding tens-of-thousands of products – at least if you are to believe the PowerPoint slides of sales projections the CEO presented last week ;-)

Now I KNOW what the first argument would be; “Don’t store in Sitecore. Sitecore is meant to build and store websites, and something as “businessey” as a Products Database has no place there”. I beg to differ however – as long as we’re not assuming there are ERP systems involved; we’re starting entirely from scratch.

I find, that actually, Sitecore is perfect for the job. Just in short summary, by using Sitecore as our data platform, we get (at the very least) the following handed to us on a silver platter:

Flexible hierarchical storage structure
Multi-lingual meta data for product descriptions and so on
Built-in advanced media library and media handling
Easily modelled data templates
Standard stuff, like workflows, security and so on
Can be edited and maintained using familiar tools

Don’t overlook this one. If you place the data in “traditional” SQL tables – YOU are going to need to write an interface that creates, edits and maintains your product data
WHAT are you going to say, when the customer asks for “advanced” stuff such as Workflows, Automatic Image Scaling / Thumbnail creation, granular (field based) security, Publishing functionality, Spell checking… ? Just naming a few here, but let’s not be blind to what Sitecore is offering out of the box
What will it cost?

So just bear with me here. Am not saying that every case is a case for data going into Sitecore and “living” there. But what I am saying is, it’s not something that should be discarded as an option without further investigation. Like everything software, there are tradeoffs involved. Make sure you make the right trade.

Setting it up

Right. So let’s get started.

In my example here, I downloaded a fresh copy of Sitecore 090416 (ZIP archive of the web root, we’re all developers here. The Installer is for marketers ;-))

I’m going to be using SQL Server Express, so I get rid of the Oracle and SQL 2000 files. For my Products Database, I will be using the Sitecore “Master” database as a foundation, so I take a copy of the files and rename them like this:

And then I proceed to attach them:

And eventually end up with 4 databases attached, like this:

So far so good. I continue to set up an IIS site for this, and a local host header of “sc090416”. All of this you hopefully know all about, so I won’t go into detail with it here. The point of this post is not basic Sitecore installation – we’re all here to look at databases ;-)

A few things that you need to do, which you wouldn’t normally, is to configure our new Products Database in Sitecore. First, open up /Website/App_Config/ConnectionStrings.config and configure the extra database. It could look like this:

<?xml version="1.0" encoding="utf-8"?>
<connectionStrings>

<add name="core" connectionString="user id=sa;password=removed;Data Source=.\SQLEXPRESS;Database=sc090416_Core" />
<add name="master" connectionString="user id=sa;password=removed;Data Source=.\SQLEXPRESS;Database=sc090416_Master" />
<add name="web" connectionString="user id=sa;password=removed;Data Source=.\SQLEXPRESS;Database=sc090416_Web" />

<add name="products" connectionString="user id=sa;password=removed;Data Source=.\SQLEXPRESS;Database=sc090416_Products" />
</connectionStrings>

Very straight forward, so far. But we’re not done yet. Open up Web.Config, look for the <databases> element, and find <!—master —>. For now, just copy the entire section – like this:

!      <!-- products –>
!      <database id="products" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
        <param desc="name">$(id)</param>
        <icon>People/16x16/cubes_blue.png</icon>
        <dataProviders hint="list:AddDataProvider">
          <dataProvider ref="dataProviders/main" param1="$(id)">
            <prefetch hint="raw:AddPrefetch">
              <sc.include file="/App_Config/Prefetch/Common.config" />
              <sc.include file="/App_Config/Prefetch/Master.config" />
            </prefetch>
          </dataProvider>
        </dataProviders>
        <securityEnabled>true</securityEnabled>
        <proxiesEnabled>false</proxiesEnabled>
        <publishVirtualItems>true</publishVirtualItems>
        <proxyDataProvider ref="proxyDataProviders/main" param1="$(id)" />
        <workflowProvider hint="defer" type="Sitecore.Workflows.Simple.WorkflowProvider, Sitecore.Kernel">
          <param desc="database">$(id)</param>
          <param desc="history store" ref="workflowHistoryStores/main" param1="$(id)" />
        </workflowProvider>
        <indexes hint="list:AddIndex">
          <index path="indexes/index[@id='system']" />
        </indexes>
        <archives hint="raw:AddArchive">
          <archive name="archive" />
          <archive name="recyclebin" />
        </archives>
        <Engines.HistoryEngine.Storage>
          <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.Kernel">
            <param connectionStringName="$(id)" />
            <EntryLifeTime>30.00:00:00</EntryLifeTime>
          </obj>
        </Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
        <cacheSizes hint="setting">
          <data>20MB</data>
          <items>10MB</items>
          <paths>500KB</paths>
          <standardValues>500KB</standardValues>
        </cacheSizes>
      </database>

Right. The only changes I made to this copy, are marked on the lines with !. Essentially the only thing changing are references to “master” which now become “products”.

With this change, I am now ready to log into Sitecore for the first time and check that everything is in order.

So far, everything is looking good. Sitecore has recognised my new database. I can switch to it – and you know… it looks just like the “master” database ;-) At this point, this should not really be a surprise.

Testing it

To further test things, I create a couple of content items. In the “master” database, I delete the /Home node, and create:

I then switch to the “products” database, and create a similar (yet different) folder.

Time to stop for a minute. Why did I delete /Home?

Well here’s the thing. The Home node that “master” is "born” with, so to speak, is just a placeholder really. At least that’s how I see it. Right now, my concern is, that if we leave the /Home node in both databases – we will have two items in two different databases, but sharing the same ID. What happens if you edit it in one database – should it overwrite changes done in the other? While pursuing this question could be fun – I don’t really think this is a scenario Sitecore will support and I frankly don’t know what would happen. At this point I don’t much care to find out either :P

So anyway.

I have my two new folders, and I do a publish. Now at this point, there are a couple of things you would be expecting to see. Upon switching to the “web” database to have a look, I think I can pretty much guarantee that whatever you were expecting, it wasn’t this:

Well ok. To be fair, maybe it was. But of all the things I personally expected when I first tried this, this was not the result I was hoping for and certainly not expecting ;-)

So what is happening here?

I guess, the most accurate answer would be, Sitecore isn’t really designed to work like this. While the concept of multiple databases IS certainly supported – you are supposed to use Proxy items to “merge” all of the data from “extra” databases (like our Products) into the main “master” database and then publish from there.

This doesn’t answer the question however, what IS happening?

Well I started investigating, and the first thing I looked into was the publishItem pipeline. Out of the box, it looks like this:

<publishItem help="Processors should derive from Sitecore.Publishing.Pipelines.PublishItem.PublishItemProcessor">
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.RaiseProcessingEvent, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.CheckVirtualItem, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.CheckSecurity, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.DetermineAction, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.PerformAction, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.RemoveUnknownChildren, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.MoveItems, Sitecore.Kernel" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.RaiseProcessedEvent, Sitecore.Kernel" runIfAborted="true" />
  <processor type="Sitecore.Publishing.Pipelines.PublishItem.UpdateStatistics, Sitecore.Kernel" runIfAborted="true">
    <traceToLog>false</traceToLog>
  </processor>
</publishItem>

And if going by names is enough (and it is), my suspicion instantly fell on RemoveUnknownChildren. A little work with Reflector quickly reveals what one of the main purposes of this item processor is.

It essentially gets a list of child item IDs in the “source” database and removes them if they are not present in the “destination” database.

This can be tested quickly enough. Switch to “master” – run a publish and check the result. Sure enough, our “Master Database” folder is now there, alone. Swithing to “products” and running a publish gives us a new result; now the “Master Database” folder is gone, but the “Products Database” folder is present.

Curious as I am, I proceeded to disable this processor, to see what happened.

<!--<processor type="Sitecore.Publishing.Pipelines.PublishItem.RemoveUnknownChildren, Sitecore.Kernel" />-->

Result:

Voila. It looks good. At least on paper ;-)

While I am not completely comfortable with an intrusion such as this, disabling a system processor in the publishing pipeline, it at least allows me to move a bit forward on what I was aiming to achieve. If I dare to, that is.

In “master” I mock up a new template, and an item named Home, based on it:

And in Products, something similar.

And after publishing the respective databases, I get the (now) expected end result.

Pretty neat :-)

Alas however, as I mentioned above, having to modify web.config to achieve this kind of behaviour worries me. I can certainly see some advantages to this model, and I hope that at some point in the future, this will be an officially supported way to work with multiple databases. For now, the route we have to go, is via Proxy Items. They are not entirely bad either – that’s not it at all – but they seem (to me) a little less intuitive to use. Worst of all, however, they don’t hide from view the potentially thousands (see CEO presentation above) of content items being proxied in from the “products” database – I would personally prefer to be able to work like I just described here.

(In reality, there are lots of potential issues involved in this approach, and I can sort of see why Sitecore wouldn’t immediately support it)

But let’s proceed.

Configuring multiple databases using Sitecore Proxy Items

First thing I do is enable the RemoveUnknownChildren processor again. Now I’m back to a normal (and therefore supported) configuration.

First thing that needs to be done, is enabling proxies on the “master” database. Find it in web.config, and toggle the setting.

<proxiesEnabled>true</proxiesEnabled>

Then, in the Content Editor (“master” database), navigate to /sitecore/system/proxies – and add a new Proxy.

Most of the settings on the Proxy Item are fairly straight forward.

The “Source Item” field is a little bit tricky however. If you click, you get a navigation tree from your… “master” database. Not products, as one would hope. This is not news, I blogged about this in January 2006 – and the workaround is fortunately even simpler today than it was back then. I open up the View tab, switch on “Raw Values” and quickly paste my ID of the “Products Database” root folder into the field. After saving my Products Proxy, I can safely disable “Raw Values” again, and now I have:

Because of what appears to be a slight quirk in the Sitecore Content Editor interface, I disable and then re-enable proxies using the new option that has now appeared in my database selector.

Once done, my content tree looks like I expect:

Notice how the items coming in from the “products” database are shown in grey. This is a visual cue to the editor, that these items are “different” – in effect in this case not coming from the same database at all.

Running a publish also yields the same results – we are now back to where we were using my first approach.

Setting up a shortcut to Products

One of the last things you would probably want to do, is set up an application shortcut to your “products” database.

Fortunately, this is very easily achieved. Switch to the “core” database, and find /sitecore/content/documents and settings/all users/start menu/left/content editor – make a duplicate of it, and name your new item Product Editor.

Configure parameters like this:

Especially make note of the “Parameters” field; where I am instructing the Content Editor application to use the “products” database instead of the default database.

Switch back to “master”, and you now have an extra option available on your Start Menu.

And clicking the new “Product Editor” now take you directly to the “products” database, ready to edit. Since this application shortcut can be configured with security just like you would expect, you can therefore configure users who can ONLY work with the “products” and not mess around with the rest of your site.

In summary

When I set out writing this article, I had a few ideas in mind. I thought I had a “new creative” approach to handling multiple databases in Sitecore – but it turned out to be perhaps a little TOO creative ;-) The recommended approach is going via Proxy Items, and it seems like the safer way to go.

Regardless of method used, I still feel that Sitecore offers plenty of options of partitioning your data if the need arises. Performance-wise… well sure – I have no doubt you could produce a QUICKER (as in; performing faster) Product Catalogue working directly with SQL Server and Products/Categories and whatnot tables.

Just like you could absolutely create a QUICKER website, using only flat .html files ;-)

But there is a LOT to be gained by utilising the tools Sitecore makes available to us. Many of them were mentioned in the beginning of this article.

I, for one, do NOT relish the idea of having to create a full blown web based product administrative interface. Especially not late Friday afternoon. Anyone else? ;-)

Sitecore Wildcard Nodes

2009-04-17T17:05:00.001+01:00

So here I was, doing a bit of late Friday afternoon catching up on the Sitecore Forums. I come across this post, in which a fairly common question is being asked, for which there are quite a number of possible solutions.

What got my attention was the reply by Kern, who mentions something called a “wildcard item”.

This is one of those humbling experiences. Having worked with Sitecore for over 3 years now, been involved directly and indirectly with a very large number of Sitecore Solution Partners, being certified, for a while had an office space right next to where certification training is being conducted in the UK; heck even going out for Friday night beers with one of the most experienced Sitecore Trainers in the UK….

And I’ve never ever heard of “wildcard items” before.

I asked around between everyone I could find online on MSN (with Sitecore experience), and although they all did offer various guesses – none knew the answer.

Best… Kept… Secret… Ever… LOL ;-) Or maybe it just never came up.

But I find it so very useful, that I’m now going to do my bit to spreading the word. Once again, it is proof that the really true gems in the Sitecore blogosphere tend to come from Lars Nielsen’s blog.

Sitecore – Avoiding query string in dynamic URL

I’ll just bring a quick snippet from the full article here, to give you some sort of idea of what it’s all about. Lars talks at length about how you can implement and use wildcard items in XSLT, and I find it in a way “drowns” the underlying real message – how you can use wildcard items in your day-to-day solutions.

Anyway, give it a look. And have a good weekend :-)

Snippet:

Migrating data into Sitecore

2009-04-13T19:26:00.001+01:00

One question I’ve noticed gets posed ever so often on the Sitecore Forums is; “How do I migrate data from my clients previous CMS into Sitecore?” Or similar type questions, to more or less the same effect. Also commonly asked on the same occasion is; “Does Sitecore provide a tool for this?”

Just to make sure we’re all on the same page here; I’m referring to the act of importing structured data (i.e. the clients existing website, sitting in any structured repository – CMS or otherwise – and getting that migrated into Sitecore for use in the Next Generation CMS system being developed by you).

This process is also known as Data Migration, and WikiPedia defines it as follows:

“Data migration is the process of transferring data between storage types, formats, or computer systems. Data migration is usually performed programmatically to achieve an automated migration, freeing up human resources from tedious tasks. It is required when organizations or individuals change computer systems or upgrade to new systems, or when systems merge (such as when the organizations that use them undergo a merger/takeover).”

It further states:

“To achieve an effective data migration procedure, data on the old system is mapped to the new system providing a design for data extraction and data loading. The design relates old data formats to the new system's formats and requirements. Programmatic data migration may involve many phases but it minimally includes data extraction where data is read from the old system and data loading where data is written to the new system.”

And lastly:

“Changing application vendor - for instance a new CRM or ERP platform will inevitably involve substantial transformation as almost every application or suite operates on its own specific data model.”

If we boil all of this information down, we get something like “Data Migration is usually performed programmatically. To do it, a design for extracting data and loading data needs to be developed.”

And that’s where you come in.

We already have at least a partial answer to one of the above questions. Sitecore couldn’t really provide a tool that could achieve this. Data extraction would always be bespoke to the specific system being migrated from – so while you might be lucky enough to find something that would extract data from say a specific version EpiServer to a specific version of Sitecore; chances are you would still end up having to do quite a bit of the data design yourself to satisfy your business requirements.

That being said, I do believe Sitecore offers a few features to help you out.

For instance, there’s the Xml Importer.

It essentially works as a Transformation Engine. Given an XML input file (presumably produced by the system you intend to migrate from), you then write an XSLT file to transform the XML input file to a pre-defined format, which the Xml Importer will then use to create Sitecore Content Items.
This is probably only fine for the simplest of cases. In most migrations I have done, there has almost always been business requirements to perform more “advanced” features such as UrlMapping (setting up Url Redirection so that the existing links will work on the target Sitecore installation), Media Importing, category mapping to name a few. Not that you couldn’t achieve most of this using XSLT. I just think it will be very hard and unworkable.

There may or may not be similar tools available. I’ve never used them myself simply because in the vast majority of cases, a pre-built tool would not be able to achieve the requirements I was tasked to fulfil. The process of actually creating Sitecore Content Items is the least of my worries when it comes to migration.

While I can’t provide a specific solution in this post to whatever migration problem you may be facing, I will take you through a fictional scenario to help explain some of the points that needs to be addressed, and provide a few techniques that you may find useful.

For demonstration purposes, I will be importing the good old trusted Northwind database. Now before you start, I’m not saying this is neither a very good idea or anything similar – Northwind doesn’t hold web content and is therefore a very unlikely candidate for this type of migration. I have chosen it simply because many of you are already familiar with the structure of it, and it is freely available for anyone to download and experiment with.

Designing the model

To begin the process of migration, you first need to analyse the data model, and come up with an implementation that will work in your Sitecore solution.

In the example I’m going to cover, I’ve been tasked with the following:

“Migrate the products into a relevant categorised Sitecore Content Structure, making sure you preserve information about each supplier for the migrated products”.

After looking a bit at the products table definition, I conclude the following:

Each product has only 1 supplier
Each product is only defined in 1 category

This is fortunate, as it will make the migration a fairly simple task. More often than not, things will not be this well defined and simple.

So in terms of a Sitecore Content Structure, this could look like this:

/sitecore/content/Home

/sitecore/content/Home/Products

/sitecore/content/Home/Products/CategoryA
/sitecore/content/Home/Products/CategoryB
… and so on

/sitecore/content/Home/Suppliers

/sitecore/content/Home/Suppliers/SupplierA
/sitecore/content/Home/Suppliers/SupplierB
… and so on

You could of course map this in many different ways. A few of the things you should be considering when designing your structure are:

How many Sitecore Items will be created on each branch?

Sitecore recommends that no more than 100 Items are created on any one branch level.

How will the existing meta-structure be preserved?

In this case, a products category is implicitly derived from the Category node it sits under. Alternatively, as I will need to do with suppliers, you could create references from your data to point to the relevant meta data.

Will any filtering or data transformation need to take place?

Often, there are old data in the source system that you might as well filter out when doing the migration. This could be data prior to a certain date (i.e. only migrate the last 3 years worth of data) or certain categories that will not be migrated to the new system.
Are there inline links that you will need to handle? As Northwind doesn’t hold web content, this is not relevant to this example. But in most cases, you will need to RegEx through content fields and rewrite Internal Links to point to the new locations of your content within Sitecore. This topic is complex enough to deserve an entire post of it’s own, so I won’t be going into detail with this here.

Having defined this structure, I now explore the relevant Database Schemas, and come up with the following 3 Sitecore Templates that will eventually hold my migrated data. In this case I started on a blank Sitecore 6 installation (090212, but that isn’t really relevant – this migration process would not differ significantly between Sitecore 5 and Sitecore 6).

Product:

Category:

Supplier:

I also create the basic skeleton structure I defined above, and run a Smart Publish. I’ll explain why, in just a bit.

Skeleton structure:

And I am now ready to proceed to the next stage.

Extracting the data

When it comes to extracting the Northwind Data, fortunately this is almost a textbook example. The data is nice and clean, referential integrity is in place, and .NET makes it very straightforward to connect to my underlying MS SQL Server to reach the relevant tables. I know there are many boilerplate examples out there of how to easily access Northwind using LINQ and similar technologies, but since the data extraction will be so simple in this case I don’t really see a need for invoking any of them.

One toolkit I will use however, is my CorePoint.DomainObjects. Don’t see this as a shameless plug; one of the main reasons I developed the toolkit was to help in my many migration projects and it just makes the task ahead so much easier.

I set up a standard Sitecore project, and quickly mock up a few entities:

Product.cs:

using System;
using System.Data;
using CorePoint.DomainObjects.SC;
using CorePoint.DomainObjects;

namespace Website.Migration
{
    [Template("user defined/product")]
    public class Product : StandardTemplate
    {
        // A few properties that will be needed to map out the Sitecore hierarchy
        public int ProductId { get; set; }
        public int SupplierId { get; set; }
        public int CategoryId { get; set; }

        [Field("product name")]         public string ProductName { get; set; }
        [Field("quantity per unit")]    public string QuantityPerUnit { get; set; }
        [Field("unit price")]           public double UnitPrice { get; set; }
        [Field("units in stock")]       public int UnitsInStock { get; set; }
        [Field("units on order")]       public int UnitsOnOrder { get; set; }
        [Field("reorder level")]        public int ReorderLevel { get; set; }
        [Field("discontinued")]         public bool Discontinued { get; set; }
        [Field("supplier")]             public Guid Supplier { get; set; }

        public Product()
        {
        }

        public Product( DataRow row )
        {
            // For brewity, null values are not checked for. There are none in the 
            // Northwind products table
            ProductName = Convert.ToString( row[ "ProductName" ] );
            SupplierId = Convert.ToInt32( row[ "SupplierID" ] );
            CategoryId = Convert.ToInt32( row[ "CategoryID" ] );
            QuantityPerUnit = Convert.ToString( row[ "QuantityPerUnit" ] );
            UnitPrice = Convert.ToDouble( row[ "UnitPrice" ] );
            UnitsInStock = Convert.ToInt32( row[ "UnitsInStock" ] );
            UnitsOnOrder = Convert.ToInt32( row[ "UnitsOnOrder" ] );
            ReorderLevel = Convert.ToInt32( row[ "ReorderLevel" ] );
            Discontinued = Convert.ToBoolean( row[ "Discontinued" ] );
        }
    }
}

Category.cs:

using System;
using System.Data;
using CorePoint.DomainObjects.SC;
using CorePoint.DomainObjects;

namespace Website.Migration
{
    [Template("user defined/category")]
    public class Category : StandardTemplate
    {
        public int CategoryId { get; set; }

        [Field("category name")]    public string CategoryName { get; set; }
        [Field("description")]      public string Description { get; set; }

        public Category()
        {
        }

        public Category( DataRow row )
        {
            CategoryId = Convert.ToInt32( row[ "CategoryID" ] );
            CategoryName = Convert.ToString( row[ "CategoryName" ] );
            Description = Convert.ToString( row[ "Description" ] );
        }
    }
}

Supplier.cs:

using System;
using System.Data;
using CorePoint.DomainObjects.SC;
using CorePoint.DomainObjects;

namespace Website.Migration
{
    [Template("user defined/supplier")]
    public class Supplier : StandardTemplate
    {
        public int SupplierId { get; set; }

        [Field("company name")]        public string CompanyName { get; set; }

        // For brewity, the rest of the fields are omitted, as
        // they're not really needed to demonstrate the migration

        public Supplier()
        {
        }

        public Supplier( DataRow row )
        {
            SupplierId = Convert.ToInt32( row[ "SupplierID" ] );
            CompanyName = Convert.ToString( row[ "CompanyName" ] );
        }
    }
}

And now we’re ready to get going. As you can see, I cheated a bit on Supplier.cs. This article is long enough, without me listing out 15 or so similar fields that don’t really add value to the demonstration / example itself.

I also created a completely standard .ASPX page under the root of the site, DoMigrate.aspx. It is in this page I will be executing the migration itself.

I start by adding some code that will load up all of the required source data into memory. When developing a migration, there are a few things to keep in mind:

Your code is more or less fire and forget. It will not run in a live environment, it will not need to be maintained by yourself or your peers for an extended period of time (normally) and therefore many of the rules and guidelines that apply to developing web applications don’t really apply here.
For performance reasons, load up as much as you possibly can into RAM. Run 64 bit if you can, and don’t worry if your migration script is eating up gigabytes of memory. If your migration mapping is complex (this example isn’t), you will appreciate the performance gain from having in-memory tables and lookups, and can in many cases mean a difference of 4 hours versus 4 minutes of execution time.
Don’t expect to get it spot-on the first time. You will be running the script many times over.

I add a bit of code to the Page_Load method:

protected void Page_Load( object sender, EventArgs e )
{
    // First we start by loading all source data into memory.

    string connectionString = 
        @"user id=sa;password=removed;Data Source=.\SQLEXPRESS;Database=Northwind";

    DataTable products = new DataTable();
    DataTable suppliers = new DataTable();
    DataTable categories = new DataTable();

    using ( var conn = new SqlConnection( connectionString ) )
    {
        using ( var da = new SqlDataAdapter( "SELECT * FROM Products", conn ) )
        {
            da.Fill( products );
        }

        using ( var da = new SqlDataAdapter( "SELECT * FROM Categories", conn ) )
        {
            da.Fill( categories );
        }

        using ( var da = new SqlDataAdapter( "SELECT * FROM Suppliers", conn ) )
        {
            da.Fill( suppliers );
        }
    }
}

As you can see, no surprises there. I am basically filling up 3 Datatables with information from the Northwind database.

I add a little code to load up all the data in the entities I created above, like this:

List<Product> ProductList = new List<Product>();
List<Category> CategoryList = new List<Category>();
List<Supplier> SupplierList = new List<Supplier>();

foreach( DataRow row in products.Rows )
    ProductList.Add( new Product( row ) );
foreach ( DataRow row in categories.Rows )
    CategoryList.Add( new Category( row ) );
foreach ( DataRow row in suppliers.Rows )
    SupplierList.Add( new Supplier( row ) );

Response.Write( string.Format( "{0} products, {1} categories, {2} suppliers<br />",
                ProductList.Count,
                CategoryList.Count,
                SupplierList.Count ) );

And for the first time, I run my migration script-to-be.

“77 products, 8 categories, 29 suppliers”

This is a good time to sanity check your code and make sure the numbers you are getting are more or less what you expect. Comparing with the rowcounts of the source database, I conclude that everything appears to be in order, and continue. It is now time to create some Sitecore Items based on this data.

Before I do that, there are a few things to point out:

The order which you create the Sitecore items in, is not irrelevant. In our case here, Products refer Suppliers and sit beneath Categories. Therefore the first Sitecore Items you create must be Suppliers (or Categories, in this case that won’t matter) followed by Categories and finally followed by Products.
I always run test migrations against the “web” database. This has a few advantages:

It runs faster (I think… it should, since there is no publishing queue handling)
It allows you to easily undo a failed migration. Just publish your skeleton again, and the whole thing resets back to start

I add some code to create Sitecore Content. Don’t worry, we’re almost done now :-)

// Set up a director for "web", standard language and security disabled
SCDirector director = new SCDirector( "web", "en", true );
var suppliersRoot = director.GetObjectByIdentifier( "/sitecore/content/home/suppliers" );
var productsRoot = director.GetObjectByIdentifier( "/sitecore/content/home/products" );

// And with these references in place, it's time to create our Sitecore Content Items

SupplierList.ForEach( delegate( Supplier s )
{
    s.Name = s.CompanyName;        // A name for the Sitecore Item (duh.. :P)
    s.ParentId = suppliersRoot.Id;
    s.Director = director;
    s.Store();
} );

CategoryList.ForEach( delegate( Category c )
{
    // First create the category
    c.Name = c.CategoryName;
    c.ParentId = productsRoot.Id;
    c.Director = director;
    c.Store();

    // And now create all products under the category
    ProductList.FindAll( p => p.CategoryId == c.CategoryId ).ForEach( delegate( Product p )
    {
        p.Name = p.ProductName;
        p.ParentId = c.Id; // .Id was assigned when the category was stored
        p.Director = director;

        // Pay attention to this one. I now map the Supplier Id field with the
        // Sitecore ID of the related supplier (created above)
        p.Supplier = SupplierList.Find( s => s.SupplierId == p.SupplierId ).Id;
        p.Store();
    } );
} );

Response.Write( "All done!<br />" );

And here we go. The data is now migrated. On my overworked laptop, this code executes in just a couple of seconds. The results look like this:

And you’re all done.

A few last things to consider are this:

If migrating large quantities of data, try and disable as many Sitecore event handlers and whatever else you can get away with.
Use BulkUpdateContext()
Don’t forget your target language
If you can, make the fields shared and unversioned. This should help migration execution speed.
And probably most importantly: Most of your work will almost always be during analysis. Figuring out

How to get to the source data
Determining what data needs to be migrated
Designing a reasonable way to store the data in Sitecore

I don’t know of many shortcuts. This is very rarely as easy a task as this one was. I know there are standards in the making, for how CMS systems could store (or at least interchange data), but I don’t know if those are realistic and (more importantly) will be widely adopted by the various CMS vendors.

I hope you found this post useful. Probably my longest post ever. Comments and feedback welcomed :-)

DomainObjects Update 3 released

2009-04-08T14:39:00.002+01:00

Yet another update was just released to the Sitecore Shared Source space.

Before I go into detail, let me just pre-warn that this is the first blog post on this blog where I’m using Windows Live Writer (thanks to John West for pointing my attention towards this tool) – and in case everything goes all wrong, this would (or could) be why ;-)

So what’s new?

A bug was fixed, where creating multiple language versions of the same DomainObject wouldn’t get correctly placed in Sitecore. Additional Language versions would get added to the current Context Language instead.
Support for Sitecore Query was added. I am a little bit torn on this functionality and whether it “fits” with the DomainObjects mindset or not. However – as there is little in way of alternatives right now – I decided to include it.
Support for Sitecore Media Items is now part of the package. As most of you will know, Media Items in Sitecore are (almost) like any other Items, yet there are subtle differences. Getting the Url of a Media Item for instance, takes you via the MediaManager instead of the LinkManager. With this implementation, users of the DomainObject are shielded from such implementation details.

How to use it

Probably best shown with a few example from the Unit Tests written for this release:

StandardTemplate item = director.QuerySingleObject( "/sitecore/content//*[@IntegerTestField='10']" );
Assert.IsNotNull( item );
Assert.IsTrue( item is TestTemplate );

item = director.QuerySingleObject<TestTemplate>( "/sitecore/content//*[@IntegerTestField='10']" );
Assert.IsNotNull( item );

item = director.QuerySingleObject<StandardTemplate>( "/sitecore/content//*[@IntegerTestField='10']" );
Assert.IsNotNull( item );

item = director.QuerySingleObject<StandardTemplate>( "/sitecore/content/home" );
Assert.IsNotNull( item );

item = director.QuerySingleObject<TestTemplate>( "/sitecore/content/home" );
Assert.IsNull( item );

There is, as one would probably expect, a similar method for querying multiple objects. Query gets passed straight through to the Sitecore API, so same rules apply as would any other Sitecore Query.

As for Media Items, the following classes have been added – all mapping to equivalent templates in Sitecore:

Audio, VersionedAudio
Doc, VersionedDoc
Document, VersionedDocument
Docx, VersionedDocx
File, VersionedFile
Flash, VersionedFlash
Image, VersionedImage
Jpeg, VersionedJpeg
MediaFolder
Movie, VersionedMovie
MP3, VersionedMP3
PDF, VersionedPDF
Zip, VersionedZip

They all inherit from StandardTemplate and mostly only differ from the fact that their .Url property is gained in a different manner.

Creating a new Media Item therefore is no different than any other DomainObject, except that the media file itself needs to be attached. Something like this:

SCDirector director = new SCDirector();
Jpeg logoFile = new Jpeg();
logoFile.CreateFromFilePath = @"C:\Logo.jpg";
logoFile.Title = "New Company Logo";
logoFile.Alt = "Company Logo";
logoFile.Description = "The new logo we received from Big Media Inc";
logoFile.ParentId = director.GetObjectByIdentifier( "/sitecore/media library" ).Id;
logoFile.Name = "Logo";
logoFile.Director = director;
logoFile.Store();

That’s it. Enjoy :-)

Old and trusted technology...

2009-04-01T14:17:00.003+01:00

A friend of mine made me aware of an ongoing initiative today. It's one of those "of course. Why didn't I think of that?" things that are found in many corners around the Web, but of which only few gets truly wide acceptance.

I'll do my bit to promote this initiative however.

"Back in the day", when I was most heavily involved in what I term "Web Client Development" or "Front End Development", IE 4.0 was "the thing". Having blown the competition out of the water (by what means, is not a discussion I get much into), finally we were seeing the full potential of the web client get unlocked. Fancy javascript, alphaFilter direct X attributes, old fashioned TABLE layouts - and best of all, there simply WERE no other browsers on the market with any significant market share. Cross Browser testing was something you'd specify as an item on any proposal, and make sure the price tag was high enough that any client would opt out.

Then along came a few... mishaps. IE 5.0 and 5.5 - none of them ever really took off. Firefox was released, and ol' Microsoft HAD to come up with something to compete again. And not a moment too soon. The answer came, in form of Internet Explorer 6.0. Or IE6 as it is lovingly coined today. According to my statistics on this blog, it is still being used by more than 35% of all readers of this blog - making it one of the largest players in the Browser Marketplace still. Not bad, considering it has been around since 2001. 8 years. What's that... two Internet Lifetimes?

Of course, in THIS day and age, everybody wants a piece of IE6 sunshine. New versions of Firefox has been released, little brothers were spawned in form of IE7 and now IE8, and even Google Chrome has entered the arena. This is not even mentioning the preferred Safari browser on the Mac platform, or perhaps even Opera.

But I say, forget all that. IE6 is a proven trusty workhorse. It will render more or less ANYTHING, and not be concerned with making your life miserable with words such as "Compliance", "Standards", "Reliability". It won't make your users confused with such advanced concepts as "Tabs". It has more security related updates than perhaps any other product in the world, and therefore MUST be the most secure product in the world.

And best of all, everyone working in IT Operations don't have to figure out how to work the Internet Explorer Administration Kit to install a later version. IE6 is already there, it's free of charge, it's integrated in the platform. What more could anyone want?

I fully back this initiative. Save IE6. Everything else just adds to an already complex environment, and trying to conform to every Tom, Dick and Harry browser product around is just a waste of everyone's time.

So say we all!

(at least today, we do ;-))

EDIT: As some of you guessed (I would have thought all, but that was not the case), this post was inspired by all the April Fool pages being posted everywhere that day – where SaveIE6 is among them. Now back to more serious matters… ;-)

090317 is out

2009-03-24T18:52:00.003Z

Also known as 6.0.1 Update-1. Looking through the change log, this one doesn't appear to be critically important. Unless of course, you are experiencing one of the issues that were fixed. On a more humourous note; a new feature arrived in this release. It reads: "A new MaxItemNameLength web.config setting was introduced, making it possible to change the maximum length of item names. The default value is 100 characters.". If I have ever publicly stated that Sitecore never listens, I hereby withdraw my statement. I blogged about this oddity only 3 years ago... ;-) And in case anybody is in doubt, I am entirely kidding :-) What I failed to realise when I blogged about this issue one spring morning back in 2006 is, that item names with more than 100 characters probably don't make much sense from a usability perspective. Why Sitecore decided to implement this as configurable NOW, I don't really know. Edit: On reading through the original post, I came across a comment from Krystle dated 13 August 2008, which mentions the MaxItemNameLength setting in web.config. Now if Krystle knew this in August, did everyone else just happen to forget about it until today where Sitecore publishes this as a "new feature" as of 24 March 2009. Lol. I am probably overworked ;-)

Surface PCs and you

2009-03-12T00:20:00.002Z

This post falls slightly (well... quite a bit, to be honest :P) off topic. But outsite the Sitecore realm of computing my eyes are very clearly focused on what I personally believe to be a big thing in the years to come; Surface Computing. Or - at least - that's what the Microsoft term for it is. Now I know we're all gobsmacked about the Sitecore 6 Page Editor facilities. But try and take a look at this, and tell me (honestly) you couldn't see technology like this implemented in a system like Sitecore. Jeff Han: Unveiling the genius of multi-touch interface design While I'm not even at a point (yet) where I can get my hands on the Microsoft Surface SDK, I know myself well enough to realise - once this technology becomes more widely available, I won't be able to keep my nose out of it. Don't take that literally ;-) Could you imagine working with the structure of your site in this manner? Organising your Sitecore Media Library? Drag-and-dropping renderings onto your layouts? I'm rambling, I'm sure. But ideas like this is what drives great development. Anyone else? :-)

Integrated Pipeline Update

2009-03-06T16:21:00.002Z

Bit busy, so just a very quick update. The Integrated Session Fix I wrote about a couple of weeks back has now passed Sitecore QA and is now made available on SDN. From personal experience I can concur, I've been running it myself on the websites I am working on and not had any problems (other than those already known and mentioned on SDN, specifically that IIS would recycle my pool often if not moving the Index folder out of the Web Root). I fully realise that me running it on a handful of non-live sites doesn't qualify as QA however, so I'm glad to now see this somewhat officially endorsed :-) The original post has also been updated.

PCheX this out...

2009-02-23T12:28:00.003Z

If you're already maintaining a decent Sitecore Blogroll you know this. But in case this blog has escaped your attention, I think you need to take a look at this: Check your packages with PCheX As someone who often designs update packages and mail them off to clients, I am sure this tool will quickly make it to the top of my list of favourite tools and save me a lot of hassle. The description says it all lol... ;-) "In a way, you can imagine PCheX to be like a little friendly elf who comes around and kindly whacks you around the head with a big rubber hammer if you design an unhealthy package. " And we certainly don't want to get whacked by big rubber hammers do we? :P

Two for the bookmarks

2009-02-18T10:44:00.002Z

Some of this information has been available for a while, it just hasn't been easy to locate. It actually still isn't ;-) But at least these two links look like they might be more or less permanent. 1) Sitecore 6 Release Notes Gives you all information about known current issues. A big help, which should help you avoid getting too stuck trying to solve a particular issue - only to find that it is already known, and in some cases even comes with a workaround. 2) Sitecore 6 Change Log Detailed information about the individual updates, so you can now be confident when choosing whether to go with the Recommended Release or the Latest Release.

DomainObjects Update 2 released

2009-02-14T18:34:00.002Z

Just posted an update of DomainObjects onto the Sitecore Shared Source space.

Apart from a few tuneups, this update is primarily about built-in support for referrers.

Say you have a product hierarchy like this:

We are then often tasked with listing or in other ways extracting products based on a selected category (or media, as could be the case in this example). In the simplest of forms, the output could look like this (although I hope no live site would ever look like this :P):

With DomainObjects this is now a very straight forward piece of code. All based on an article by Lars Floe Nielsen which I often refer back to.

Sitecore 6 package, sources, API documentation have all been posted to the Trac space.

Wow @ Sitecore 6 Item Creation Speed

2009-02-10T14:14:00.003Z

I remember this was being brought up right around the time of release of Sitecore 6, that item creation speed had been increased since Sitecore 5. To be honest, I haven't really noticed. Until today that is. For a client, I am currently in the process of converting/migrating an entire website based on the Polopoly CMS system to Sitecore 6. Essentially a process of getting all the source content, parsing and understanding it, mapping it to relevant Sitecore templates (around 20) in all and writing this entire new structure into Sitecore. Take a look at the numbers I am getting. Migration process starting.... Deleting existing content....done! Building Table of Contents tree....done! Building Departments tree....done! Performing in-memory mapping...done! Duration: -00:03:58.2400000 7271 items created in total - 471 products - 59 images - 1678 documents - 3728 other articles - 1335 departments 7.271 items created (using BulkUpdateContext()) in just under 4 minutes. One word. Fast! :-)

Session State and the Integrated Pipeline in Sitecore

2009-02-05T21:00:00.006Z

Currently (release 090120) Sitecore has a known issue when it comes to running in the new IIS7 Integrated Mode pipeline. The issue states: "The ASP.NET session state may be unavailable to web page’s code depending on the URL used to access a Web site’s homepage. This occurs with a short URL, such as http://mysite.com/, while longer URLs, such as http://mysite.com/default.aspx do not encounter the issue. " It actually goes a bit further than that. I did some testing on a completely fresh 090120, Created "Test1" under "Home". And then "Test2" under "Test1". Like so. Sitecore - Content - - Home - - - Test1 - - - - Test2 I set up my website to run 32 bit Integrated Pipeline, with a binding to host header "integrated". I then added some test code to the Sample Layout.aspx, like so. % string lala; if ( HttpContext.Current.Session != null ) lala = "Session has value"; else lala = "Session is NULL"; Response.Write( lala ); % (Extremely bad coding practice however, don't do this in any real scenario :P Breaks preview and page editor and whatnot... was just to test) Now, after navigating to the following URLs, I was getting these results. http://integrated/ "Session is NULL" http://integrated/default.aspx "Session has value" http://integrated/test1 "Session is NULL" http://integrated/test1.aspx "Session has value" Ok so. A known issue and according to Sitecore Support, no immediate ETA for a fix. Why would you want this however? Well let's consider SEO. Often have I been met with the requirement that SEO friendly URLs are required. And for some reason, SEO specialists seem to think that http://integrated/products/classical-piano is a better URL than http://integrated/products/classical-piano.aspx. Whether they are right or not, I leave with someone who knows more about these things than I do. Bottom line is, I needed these types of URLs for a client of mine. And I was curious :P According to Microsoft, the list of Breaking Changes for Integrated Pipeline isn't all that long, and none of those stood out to me as something that should be affecting Sitecore. After much digging around, searching the web and reading lots and lots of articles about this new Integrated Pipeline - many of which was full of praise but none really helping me in any way - I finally came across something from someone who knew "what's what". ASP.NET Session State for Native HTTP Requests in IIS 7.0 Upon reading, I felt 99% sure this was the fix I was looking for. A bit of coding, and a bit of optimisation (read through the comments) and I was ready to give it a shot. And it worked :-) A bit of coding (or rather, code pasting), a compile, a simple web.config change and voila. Now all of the above URLs seem perfectly happy to give me the Session State I've come to rely on. If you can't be bothered to set up a project for this yourself, I'm attaching the project and the DLL. Just plop the DLL into your /bin folder and add the following to your web.config at the end of the system.webserver/modules node. add name="CorePointIntegratedSessionFix" type="CorePoint.IntegratedSessionFix.Module, CorePoint.IntegratedSessionFix" remove name="Session" add name="Session" type="System.Web.SessionState.SessionStateModule" preCondition="" (excuse the markup, I am REALLY not happy with my blog texteditor.. but that's a story for another day) And the project file here: CorePoint.IntegratedSessionFix.rar Enjoy :-) Caveat: Be aware however, I don't have the resources to execute a complete regression test of this fix. It seems fine, but there could be repercussions that I am not seeing at this stage. I have an open support ticket with Sitecore Support on this issue; and I will post these findings there for their review. Will update if needed. Update 23-February-2009 While eating my own dogfood on this fix, I was experiencing some fairly odd behaviour when using the Sitecore Content Editor. It would pause ever so often for a few seconds, and sometimes operations would get aborted and throw up a dialog box like "Session Terminated Unexpectedly". Quite naturally I was worried this had to do with this fix. Turns out, it hasn't. It IS related to the integrated pipeline however - I had overlooked another issue from the Known Issues List which states: A known issue in Microsoft IIS may cause Sitecore to restart during index rebuilding or package installation when Sitecore indexes are stored below the Sitecore web site root folder.Moving the Sitecore indexes outside website root will resolve the issue.To configure this, add the following to the "settings" section of web.config file: setting name="IndexFolder" value="C:\test\indexes" Note: You must ensure that the application pool identity has full control over the chosen index folder. For more information about appropriate folder permissions, please refer to the Sitecore CMS 6 Installation Guide. So if you do run integrated, don't get caught by this one. Like I just did :-) Update: 6-March-2009 This has now passed Sitecore QA and is an official workaround according to SDN.

Expert Systems

2009-02-03T16:45:00.002Z

I know many of you follow this blog, but today's entry certainly did bring back memories. Those of us who worked when the IT industry (or maybe just the Internet industry, I am not THAT old :P) fired off probably all have some terrifying tale to tell that resembles todays entry in TheDailyWTF. Be it some director's nephew "who was a genuine whizkid with websites", old-skool Mainframe programmers adjusting to the new wave of computing or whatnot. Fortunately, these days, this phenomenon is pretty much a thing of the past. Mostly anyway. IT/Web projects have become business critical and are therefore generally regarded as something best left to proven professionals with at least something of a track record. But even professionals get it wrong. If we look at the article, Victor is probably what WTF describes as "A Complicator". In summary, described as: "There are some so deeply embedded within The Complicators, that they've acquired a sort of sixth-sense: the ability to find meta-problems ("a problem with the process of creating a solution for the actual problem") in virtually any solution. As we've all seen, the systems that these developers create often end up as a barely functional application comprised of a Matryoshka-doll-like nesting of problems and solutions." And this would all be funny if the most recent experience I had with such a type was "back in the 90s". It was actually very recent. As a Sitecore consultant (as in, a consultant working with the Sitecore product, not to be confused with a consultant working for Sitecore) I was recently tasked with what looked to be a very straightforward assignment. The client had oversold (which is impressive given the current state of financial affairs) and needed an extra pair of hands to wrap up some loose ends on a few existing projects and then help on the implementation of 1-2 new sites from scratch. The client didn't have years and years of experience in working with Sitecore, and therefore already had another professional Sitecore consultant on the task. He apparently came with an impressive track record, a good friend of the house and so on. All in all, this was promising to be a sweet deal and I was looking forward to it. The first week or so, I would be working on my own. No problem. I mean, that's what one would be thinking, no? Almost complete site, just a few issues with some "Related People" not showing quite like the client wanted it. Sounded like straight forward filtering to me, or possibly adjusting a few queries. I should know better by now, that to jump to conclusions ;-) Much to my surprise, when I started digging in, I found a Sitecore implementation unlike anything I've ever seen before. And I've seen quite a few. Both done in companies I worked for, in clients I've contracted to, in solutions I've been tasked to review. I was confident in my belief that I pretty much had this thing called "Sitecore" nailed and nothing could truly surprise me to any real extent. Well how about this then. Ever seen a entire Sitecore site done in just 1 rendering sitting hardcoded into 1 layout? Ever seen a Sitecore site with 3 layers of inherited templates (65 of them in all, just about) yet the ENTIRE site consisted of content items ALL based on just 1 "Mega Template" (aptly named "Page")? And being designed like this, ever tried to explain to your end user why there are 50 fields, and depending on what values you define in some of them, other fields need to have specific values set? Complicator indeed. We're not talking a simple website here either, mind you. While it is difficult to rate the complexity of a website, it had all the bits you would normally expect today. Content pages, menu structure, left hand navigation, right hand spot elements, you know. The works. Ok so. With the issue at hand; "Mr. So and So displays BELOW Mrs. This and That on the People Page. We need the order to be reversed" I started investigating. Now what probably should have been a 30 minute task ended up taking a couple of days of my time. Backtracking through an immense hierarchy of included XSLT renderings, mode="whatIneedNow" switches and so on - just to get a basic overview of how this Expert System was actually producing page output. Must be my limited mental capacity shining through, or me getting old or something. Or both ;-) See to build a site like this, essentially bypassing everything "Sitecore" except for perhaps "Sitecore-as-an-advanced-field-editor", one needs to come up with an alternative system. Enter "The Meta System". A sophisticated hierarchy of tags, metatags and content "blobs" which users could then (in theory, at least) tag up and ultimately have displayed on the site. See there was a "PeopleLister" meta component, an "Office" component, and if you happened to tag everything up just right - you would eventually get information out on your site. All generated by the very clever rendering Expert System. And as for the problem at hand, I guess the system didn't really consider the order People needed to be presented. Worse yet - this changed from time to time. Sometimes an alphabetical list of All People was needed, and sometimes the order (as in the case I was tracking) followed "silly business rules" (this is an actual quote). A week came and went, and I finally had the chance to confront my colleague-consultant. While my first inclination was to ask "WTF!?" I did start on a more subtle approach. After all, being a visitor to the client a consultant is expected to show some degree of humility - maybe they preferred it their way. Turns out they did. Q: But what about the Page Editor? Surely clients must have seen a Sitecore demonstration, Nicam or otherwise, and expect this to work? A: Nah, noone uses that anyway. It's not imporant. Q: Ok. So what about caching? Sitecore has little chance to implement much in terms of useful caching when all it sees is ONE rendering on ONE layout? A: That's not important. Servers are so quick today anyway. Q: Umm ok. Well then what about user security? How will you assign permissions in this hierarchy of metadata - controlling which users can actually access/change/delete what? A: We just give everyone "Is Administrator" permissions and write a guide. If they break anything we can always restore back. Q: Aha. Well how about integrating other Sitecore modules, like for eCommerce, SEO and so on? A: It can be done. Or we can just install those on a separate installation and run it on a sub domain. That's what the users want anyway, not to complicate things. Some may think I am kidding, sadly this is not the case :P Not to complicate things.... Sitecore Support, you have my heartfelt sympathies. I certainly hope you don't get many of these cases dumped into your support tool. And if you do, kudos for what must be endlessly much more patience than I have myself. After these initial rounds in the ring, everyone decided it was probably best I went on to greener pastures and left them to it ;-) Let me end this by pointing out what I have already pointed out in the past; please DO be aware that Sitecore is putting great effort into publishing guidelines, best practices and "Cookbooks" for how to approach Sitecore site implementation, development and handling. They're all on SDN. I haven't had time to read through ALL of them yet (shame on me); but if anyone has - is there a definition for Expert System in there? ;-)

Temporary files, again....

2009-01-28T12:10:00.002Z

So about a month and a half after blogging about this very issue, here we go again. This time I wasn't thinking very deeply about things, so when I tried to install Update 8 and got hit by a File Exists error I contacted Sitecore Support.

Luckily these guys are not without humour, and referred me back to my own blog post for a fix LOL ;-) 1 : 0 to Sitecore Support.

As can be seen, apparently all of these files were created 14/1. Now I have NO idea what I was doing that particular day, but I was wondering.... anyone else having this problem or is it just something really messed up on my development box here?

Gonna get to the bottom of this, one day... :P

typesThatShouldNotBeExpanded

2009-01-26T14:34:00.002Z

After having ran into this myself a few times, and for the 5th time being asked by one of my Sitecore friends; I though I would bring this one up. Ever had trouble getting your ASP.NET ListView controls to retain ViewState or do standard Paging? As many of you will know, Sitecore's web.config is a source of endless configuration fun. Sitecore's modular nature is basically founded on this configuration - want to switch something around or change how Sitecore works (at your own risk, usually, but that's ok)? Web.config is the place to go. Many of the settings are self explanatory and well commented, so working out what they do is fairly straight forward. In some cases, they are not, however. If you look through, you will find an element named " typesThatShouldNotBeExpanded". In it you will find two entries:

System.Web.UI.WebControls.Repeater
System.Web.UI.WebControls.DataList

And while I don't have enough insight into exactly HOW Sitecore ties in to the ASP.NET lifecycle to determine exactly WHY this setting is needed, I can however tell you what happens if you don't fully control it.

Say you have a control, where you drop in an ASP.NET ListView Control. Or a GridView control. They'll appear to work fine at first, but you'll quickly notice that paging and viewstate is acting up a bit on your controls.

The fix for this is easy, and as far as I know this is well known by Sitecore Support. Add your controls to this - almost black magic like - web.config setting, and you're set :-)

Add System.Web.UI.WebControls.ListView, and System.Web.UI.WebControls.GridView to this collection if you find paging, viewstate or other things "that normally works" doesn't.

I don't know when this setting was introduced, or even IF there is a reason Sitecore doesn't include these common controls in the list. Maybe there's a negative effect on performance or otherwise.

Searching on the web only produces this one link (searching on SDN, two results but similarly without explanation), and that doesn't tell us anything else than the config setting was apparently introduced in Sitecore 5.3.2 071220.

Sitecore MVPs announced

2009-01-06T20:07:00.010Z

While those of us nominated have known this for a while, Sitecore today officially announced the Sitecore MVPs for 2008. A select list of 13 Sitecore professionals, of which I am proud to be named as one. Reportedly I am the first (and so far the only) UK based Sitecore MVP - and given I make my living here in London as an independent Sitecore consultant, I can only hope this award helps open up a few doors down the road. Up until the nomination came through on email, I didn't actually really know if the MVP programme was "alive" or if it was just something that would be "nice to have" inside Sitecore circles. I know Alex De Groot used the title for a while (and considering he probably wrote 10 blog posts for every one of the rest of the community put together, probably rightly so :P), but other than that the title didn't come up very often. Obviously Alex works for Sitecore now, and so probably no longer qualifies as such - not that his community contributions are less noticed for that reason. Cheers Alex, for being the one nominating me for this. This brings about a point that I consider to be important. The MVP programme nominates people outside Sitecore itself. Which, with some reason, should make the MVPs a reliable source for real impressions and opinions on Sitecore as a product, Sitecore as a company and Sitecore as a business partner (well maybe less so). Without restrictions being imposed, mind you, as Alex himself commented on earlier in 2008. While personally I certainly have a vested interest in Sitecore (as I make my living from implementing it), I'd also like to think that I keep an open mind and share my thoughts on it all - be it positive or negative. No doubt this has been the source of some controversy from time to time ;-) For that reason, here is the official list of Sitecore MVPs for 2008, with links to their blogs. I may not have been able to locate you all, so if you're on this list and have a blog I have not included - drop a comment. Sitecore will eventually make a similar list available on their site but until then, this should help the interested reader. Australia · Alistair Deneys (Next Digital Group) Canada · Eric Briand (Ergonet) · Glen McInnis (Non-linear Creations) Germany · Christopher Wojciech (Netzkern GmbH) · Julius Ganns (Netzkern GmbH) Denmark · Klaus Petersen (Alpha Solutions A/S) · Thomas Eldblom (Pentia A/S) The Netherlands · Justin Sjouw (Caesar Optimit) · Marc van Aalst (Evident) The United Kingdom · Mark Cassidy (CorePoint IT Limited) (btw, please don't click the company link... yet... :P) USA · Andy Uzick (FMC Technologies, Inc.) · Sasha Pfandt (Digitaria Interactive, Inc.) · Ben Golden (Aware Web Solutions) (must be cold in the Twin Cities this time of year :P) Congrats everyone, and happy new year :-)

TreelistEx not registering links in LinkDatabase

2008-12-18T23:13:00.004Z

Ahh nothing like being kept up till 11pm on the last day before xmas holidays... :P So here's the thing. TreelistEx is broken. You're supposed to be using this field type to "tag" content. Probably most often, you will use this to tag up categories for news articles, blog posts, meta keywords and similar stuff. And on that same note, quite obviously, you're supposed to be able to "reverse lookup" all these tags you can set. Not much point in having to loop through ALL of your 100.000 articles to find the ones that are marked "Holiday Capers" - Sitecore has a Link Database for that. If you've not worked with the Link Database before, Lars Nielsen has an excellent post about it. But here's the catch. As I said, this is broken. Nothing you store in a TreelistEx field will be registered in the LinkDatabase. "tree list", reference, droplink - sure. They all work. But not this one. A little work with Dr. Reflector eventually led me in the right direction. Now why one would need to, is an entirely different story for a different day - I believe I already had a few comments on that earlier. It comes down to TreelistEx being unknown to Sitecore itself - odd as this may sound. The field type simply doesn't exist in FieldTypes.config - and this will eventually force Sitecore to skip the field when doing it's internal RebuildItem functionality. This is, as Reflector will tell you, the method that Sitecore will recursively call when rebuilding the Link Database. Add the missing field type to /App_Config/FieldTypes.config - force a full rebuild of the Link Database - and everything now clicks into place as it should. fieldType name="TreelistEx" type="Sitecore.Data.Fields.MultilistField,Sitecore.Kernel" Thanks Sitecore. Where do I bill my lack of sleep? ;-) EDIT: 14-February-2009 This fix was released in build 090212 (also known as Service Release 1).

Showing Sitecore how to improve...

2008-12-18T15:04:00.002Z

A while back, Alex De Groot invited to an open debate on areas where Sitecore could improve. This post is my response. Readers of this blog will have noticed, I often take a somewhat critical stance to Sitecore. What may not be so obvious at a first glance however is, my beef is not so much with the product itself, but rather Sitecore as the company behind the product. The issues that come from working with the product ultimately comes down to the practices that are in place in the company that creates it. Sure, a particular bug might be annoying but it's just the smoke - not the fire itself. So I won't be requesting features, in short. In fact, if anything, I will request "no new features" at all; but instead point out that good solid project management practices, quality assurance, usability studies and whatnot make up a very important (if not the most important) part of any ongoing software development endeveaour. Even if I couldn't spell that right... ;-) But then - it is also obviously very easy for me to just sit here and say "hey guys, manage your stuff - how hard can it be?!". I've been doing software development for many years now, and if anything I've come to realise that perfection is more a direction than a goal in itself. I've seen improvements in the way Sitecore has been working, especially over the last year or so. Documentation for instance, has always been a weakness when working with Sitecore. So I was very pleased to find, that documentation is taking on a whole new level with The Sitecore 6 Documentation Package - while far from covering everything there is to say about working with Sitecore, it's certainly a step in the right direction. Through Sitecore Support I've had the priviledge of looking at some of the other documents still being drafted, and I'm glad to see that this new trend is taking on. Documentation would have been top of my list still, as the first thing to improve. It doesn't matter that you have a good product, if noone knows about it. With that said however, for crying out loud guys will you fix and consolidate your online information already? Per Bering from Codehouse pointed this out as well as a comment to Alex' post. Your site search is absolute bollox and there is not a single other thing to say about it :P "Did you mean.. xsl rendering xsls rendering xslext rendering exslt rendering xslt rederings xslt renderingid xslt renderingitem xslt renderendtag " ... :P Right now, the only way I am staying "on top" (if that's what I am, I don't know) of current Sitecore technical information is, by subscribing to roughly 20 Sitecore related blogs, chatting with people who work for Sitecore, chatting with other consultants who work for Sitecore partners and then as the last resort - the ultimate Sitecore tool; Reflector. I've even seen Sitecore support personnel themselves point developers in the direction of Reflector as a source for information. And while it's a great tool, I can't really believe this is how it is meant to be. Notice btw, how I deliberately didn't include SDN5 as a common source of information for me. While I go there, it's very rare that I am able to successfully find what I'm looking for there - so the fallback option is Sitecore Support. Fortunately I find Sitecore Support very good, at least :-) Enough about documentation however. The second thing I would ask for, is for Sitecore to start eating their own dogfood. If it's not good enough for you to run your own site on, it's certainly not good enough for the very large developer base out here who has very limited access to documentation on the product. Update notes and changelogs don't really tell a useful story as I think most who work with the product will know. I've never seen a changelog saying that the "admin" user no longer comes with Danish locale as default in a blank Sitecore 6 - and that this breaks the way some fields work in Sitecore - nor have I seen a note that it was fixed. It might exist somewhere; see "searching" above ;-) Or what about Page Editor acting up and throwing exceptions if you happen to run Sitecore in a non "en" language context? Or breaks if you don't set a layout for the Print device? I would much rather wait an additional 3 months for any Sitecore release, if it means it has been fully implemented throughout the Sitecore organisation before being released to the general public. This will not cover all usage scenarios of Sitecore as a product - but it WILL get rid of "all those little annoyances" that are so very hard to explain to our customers. "Why does my image links break, if I do it this way instead of that way?" and so on. Fortunately, level 1 support is behind me - but then.. should we accept these kinds of issues at all? Finally, and just for good measure, I will throw in a product feature comment. The Page Editor. Brilliant, but hugely underestimated right now. Find a good way to solve the problems of configurable controls (a.la SharePoint WebParts) getting easily added to layouts - and come up with some form of layout or rendering inheritance as pointed out by another commentator. Not necessarily saying the suggested approach is the best one (although it very well might be), but it certainly is something that I would like to see addressed and solved. In summary:

Document. Love your developer base, please ;-) Take a look at what the competition is doing, maybe expecting a new MSDN is asking too much after all ;-)
Eat it yourselves. First.
Keep evolving the Page Editor, which I honestly believe can be one of the absolute killer features and opens up a lot of opportunities for new development areas.

That's not asking for much is it? ;-) We were invited however.