MemorableURL.com

SharePoint, Documentum or a Hybrid Approach? Capgemini's Answer.

I attended a Capgemini session today at EMC World that looked at when to use SharePoint, when to use Documentum and when to consider a hybrid solution. Jon Ludwig, the speaker, did a really good job of walking through the strengths and weaknesses of each solution, the available options and how to combine the solutions to get a best of breed solution.

Jon opened with a slide that pretty much summed up the key differences in the two company’s approaches, (included with his permission). On the left he had SharePoint which started with a UI focus and worked down to a supported repository. On the right he had Documentum starting with a repository focus and moving down to the UI – he pointed out that the original version of Documentum didn’t even have a UI. He described Documentum’s UIs are being ‘power user’ focused which is probably a fair representation.

 
He then used the slide below to show the functionality in SharePoint 2007, SharePoint 2010 and Documentum from a UI perspective.

This was interesting in itself but when you look at it next to the same comparison from a repository functionality view you really get to the meat of Jon’s key points. SharePoint is certainly making huge gains in the space of UI functionality but there’s still plenty of room at the back-end for add-ons from traditional ECM vendors.

One interesting side note was that Jon suggested that as soon as SharePoint adds specific functionality to SharePoint it then becomes a commodity which other vendors move away from. He quoted the fact that when SharePoint added enhanced WCM to SharePoint in 2010, Documentum partnered with Fatwire to move out of the WCM area and into ‘Web Experience Management’.  I’d suggest that this is the only the case when SharePoint gets close to an acceptable level of functionality – take workflow for example, I don’t see vendors moving away from real workflow/BPM/BPA…quite the contrary.

Jon danced around the issues of SharePoint’s ability to scale; he stayed on the fairly safe ground by talking about the fact that you would not want to ingest tens of thousands of images, heterogeneous data types or large objects into SharePoint.

Records Management

Jon spoke at some length about SharePoint 2010’s RM capabilities. He pointed out that SharePoint 2010 has most of the bits that you need to do records management but that SharePoint assumes that everything that is a record can be moved into SharePoint. This assumes that SharePoint can handle the scale and the types of objects that would need to flow in. He pointed out that SharePoint was not designed to be an enterprise scale repository’ in fact he specifically says “Microsoft will likely never offer full repository functionality; it’s not their focus.”

He went on to talk about federated RM as a potential solution.  With federated RM you can leave SharePoint content in SharePoint, leave file system content on the file system and manage it all using your file plan in Documentum. His story here was that you could get content into Documentum when it makes sense and the federate content that is not there. Documentum owns the file plan and the records management features but the content can stay in SharePoint (or elsewhere).

When talking about acting as a trusted advisor he made one interesting comment, he said that customers should not just do a feature/function comparison between SharePoint and the traditional ECM system and act on that alone, they need to also consider what it would cost to move everything into SharePoint. He also advised that he liked federated because he felt that you should give your records admins just one tool to use, keep it simple.

Jon was definitely a fan of using a hybrid solution for RM, not picking either one or the other solutions.

In general I think that Jon really grasped the issue at hand and had real, solid and pragmatic advice for customers. He did not bash SharePoint or Documentum, he pointed out the weaknesses and the strengths of both systems and seemed to focus on why the right solution at the right time would give you more than the sum of the parts.  

Jon can be contacted at [email protected].

The Evolution of Content Externalization

It is EMC World this week and as usual we are announcing a multitude of new/updated products. Two of them are relevant to my usual subject matter and thinking about them raised one interesting observation about the commoditization of certain aspects of data externalization from SharePoint.

The two products are ‘EMC SourceOne for Microsoft SharePoint’ and the next release of ‘Documentum Repository Services for SharePoint’. The former product is a product aimed at removing content from SharePoint and placing it in an archive; for operational reasons and for long-term preservation of SharePoint content that is no longer accessed actively from SharePoint. The latter I’ve mentioned before but it is basically a way of connecting SharePoint to Documentum in a way that is invisible to the SharePoint end user.

I’m not known for gratuitously marketing my employer’s products so you might wonder why I’m bringing this up in my Blog. Almost exactly a year ago I wrote a set of articles that focused on externalizing unstructured content from SQL Server. I concluded that there were four primary ways to handle externalized content:

1. Dump the unstructured content on to the local file system from SQL Server - this gets the content out of SharePoint and relieves any SQL Server bloat related issues but you do not get any additional value out of the practice.
2. Externalize all content types and store them in an archive alongside your other archived content - as well as solving the SQL bloat issues this adds significant value with common policy management, long term archiving and retention/disposition/litigation support on the archived content.
3. Store unstructured content in your ECM system alongside your existing content – fixes SQL bloat; provides policy enforcement, long term archiving and compliance management plus the ability to re-purpose and reuse the content. Also gives you a natural integration point in to other enterprise information systems
4. Put the content in an ECM system and use SharePoint as a portal – an excellent solution for accessing existing ECM content and processes but doesn’t really leverage SharePoint’s native capabilities.

One year on, where are we? Consider a brief analysis of each of the four options today…

1.    There are so many solutions for simple file system externalization that some vendors are giving these away for free. The premise is that rather than storing the unstructured BLOBs (Binary Large OBjects) in SQL Server you use Microsoft supported APIs (EBS/RBS) to store the BLOBs on a local file server. This reduces DB bloat and helps smooth out the IO flow.

2.    Storing your content in an archive works well for inactive content but is typically too invasive for most active content. The reason is that in this model the content is either moved to the archive and no longer a ‘first class’ SharePoint citizen or a copy is made in the archive for compliance purposes.

3.    If you need the power of a traditional ECM system and want your SharePoint content to play nicely with all of the other corporate assets then this works well. If you just want the content out of SQL Server then options #1 & #2 are probably more cost effective.

4.    As long as we have traditional ECM systems then this option will be around. Although this option can give you access to the power of the traditional ECM system you typically lose a lot of the rich functionality of native SharePoint so this is always not for everyone’s data.

One year ago you could find Microsoft partners touting solutions modeled around any of these options. However, the thing that I noticed about the SourceOne archive product and the new version of Repository services was that they both include BLOB externalization to the file system as a basic piece of their architecture; in fact I’d argue that in both cases this capability is being undersold because it is just a byproduct of the actual capability of the product.

Both products can externalize content to a file system and then optionally perform their core operations on the content. The SourceOne product externalizes content to the file system and then based on rules it can archive the same content out to a common repository to be managed alongside archive email and archive file system data, (the latter is to be released later this year I believe). In the repository services case the product has always externalized content to a staging area before it was ingested into Documentum but in the new release you can write more granular rules to leave some content in that staging area and only have specific content moved into Documentum.

I wonder which one of the remaining three will become just a product feature in another year…or maybe a new option will emerge. Perhaps the real question will be ‘when will Microsoft fill this architectural gap?’ in the future. Contrary to hype RBS does not do this in the core product; RBS simply adds another area where the externalization can occur. In fact, for solutions that need to do ‘intelligent’ management of the underlying content RBS is a much less elegant solution. More on that in a later post perhaps!

Managing Social Data: Is SharePoint the Answer?

I was approached late last year by ARMA to write an article for their Information Management publication. During a lapse of good judgment I suggested that a topic that everyone is interested in but no one is talking about is how to apply compliance controls to ‘semi-structured data’.  Semi-structured data being those things like Blogs, Wikis, activity feeds, calendar items, etc. that are stored as structured data but look very similar to unstructured data when being consumed. 

Here's the intro to the article and the cover of the magazine. Before you ask, no that's not me hiding behind the question mark.

Managing Social Data:
Is SharePoint the Answer?

Records management (RM) professionals have been challenged to manage electronic data for some time. Their efforts have tended to focus on unstructured data, such as documents, scanned images, and spreadsheets. Recently, structured data held in databases has also come under close scrutiny, especially during discovery. However, there is a third category of data that tends to be overlooked. Read on for more or download the PDF version

Although users of all semi-structured data providers need to take heed, I pointed out that SharePoint was an opportunity for companies to look at a single point to apply these controls. If you are using SharePoint for all of your ‘Web 2.0’-type content then why not look to SharePoint to apply the compliance controls that you need.

Interestingly, when I was 80% of the way through writing the document I reached the conclusion that all I was going to provide you my dear readers was the bad news; I was not unearthing much in the way of solutions to these issues. Luckily, I was able to get some time from some of the SharePoint product team and they were able to add some of the color around the capabilities of SharePoint 2010 - thanks to all concerned. I’d also like to thank the team from Gimmal who provided some of the use cases and real-life background data.

The article is long…maybe too long but it should be worth a read if you care about SharePoint, compliance or have chronic insomnia. This is a topic which will only get worse before we catch up...

Main article: http://content.arma.org/IMM/March-April2010/IMM0310managingsocialdata.aspx

PDF Version: http://content.arma.org/IMM/Libraries/March-April_2010_PDFs/IMM_0310_managing_social_data.sflb.ashx

Note: My email address is quoted incorrectly in the magazine, it is actually [email protected] (with an underscore not a period). My fault not ARMA's.

 

SharePoint 2010 Content Management

I was asked to write an article about SharePoint 2010's ECM capabilities after the SharePoint conference for SharePointPro Connections magazine. It took a while for them to publish it and given the huge amount of quality SharePoint-related information that they have I thought it might get lost so here's a copy.

I tried to stay agnostic - I know that the SharePoint guys will think that I am SharePoint bashing and the traditional ECM guys might complain that I am being too soft on Microsoft but I think that this is a fair representation of the changes. As always - comments are appreciated...

Head over to http://www.sharepointproconnections.com/ and check out all of their other articles.

---

When SharePoint 2007 first started being widely adopted, the question of whether it could be used to replace existing ECM solutions was often asked. The general consensus was that SharePoint 2007 was not necessarily the best choice for the corporate ECM backbone. However, when SharePoint 2007 is coupled correctly with a traditional ECM solution the two systems complement each other extremely well.
With the imminent arrival of SharePoint 2010 that same question is bound to arise – do we need SharePoint 2010 and our traditional ECM solutions? Let's walk through the content management-centric features of SharePoint 2010, see what works, and look for where there is still room for improvement.

SharePoint 2010’s ECM Focus
Not surprisingly SharePoint 2010 builds out in a lot of the areas that SharePoint 2007 saw success; enhancing developer-related functionality, building communities, integrating search, supporting business analytics and the ubiquitous Office integrations.

The Microsoft SharePoint development team also addressed areas related to content management. This article highlights the key content management-related features in SharePoint 2010 and questions how well they might address enterprise requirements.

Library Services: Managing Content in Large Scale Deployments
SharePoint 2010 does address some issues related to how to handle large volumes of data – the removal of some list size limitations and support for remote BLOB storage providers in SQL Server for example. However, rather than focusing primarily on the back-end scalability of the system it seems that Microsoft took a more user-centric approach to this challenge.

It would appear that they asked the question “If I have huge volumes of data how might users interact with that data?” Many of the features highlighted in this article relate to tagging, finding and interacting with specific content which supports this assertion.

Metadata Enhancements
One of the single biggest challenges in enterprise content management has always been mapping content with relevant and accurate metadata. If you accurately tag content in your ECM system there is a reasonable chance that you’ll be able to find it again later.

If you don’t, then good luck wading through hundreds of thousands of documents looking for the one you wanted. SharePoint 2010 has focused a lot of effort in this one area. They have introduced Enterprise Managed Metadata services, location-based metadata defaults, unique document IDs and content ratings.

Enterprise Managed Metadata Services
The Enterprise Managed Metadata (EMM) service allows you to create one or more centralized libraries containing content types, terms and managed keywords. These can be published to sites across your farms and also accessed via web services by third party applications.

For example, you can push object types out to all sites across your farms where they can then be either used directly or inherited from locally as base types to create custom types. Note that keywords and terms don’t use the push model as they are used on-demand so they’re always current.

Included in the managed metadata are taxonomies (a formally managed hierarchy of terms) and folksonomies (unmanaged user defined terms). Interestingly, users can not only create folksonomy terms but they can also have those terms pushed out to the owner of a specific taxonomy for potential inclusion in the managed list.

If I were a corporate librarian, I might find this pathologically annoying or amazingly useful – I’m not sure which. If you fill out a field that is related to a taxonomy you get type-ahead assistance which not only makes it easier to enter values but it also cuts down on the number of erroneous errors that one tends to get in a folksonomy-based system.

Location-Based Metadata Defaults
This feature allows you to automatically set default values for metadata when an object is uploaded to a folder or a document set. For example, all documents in the ‘HR Contracts’ folder could have the ‘Department’ column set to ‘HR’ and the ‘Document Type’ set to ‘Contract’. These rules cascade down the folder structure which helps too.

Given that SharePoint 2010 is promoting a ‘navigate using metadata’ paradigm rather than a folder model this feature is very useful. For now, users could continue to use folders as their ‘filing system’ but behind the scenes the content is being discretely tagged.

Unique Document IDs
The idea that SharePoint 2007 did not have unique document IDs seems beyond comprehension. How was SharePoint able to track which object was which without this fundamental identifier? It turns out that SharePoint 2007 did use unique identifiers but only internally.

SharePoint 2010 externalizes the use of unique IDs. Although you can enter these IDs directly in to the UI and bring up a document they are better suited to being used either programmatically or embedded in hypertext links; here’s an example of one just so you understand why: 7K3W6YVEA2YC-17-1. You can define naming schemes so a document can be identified even if it moves between site collections.

One thing missing from the unique IDs is that each version of the document does not have its own ID. Ideally you would want to be able to address an individual document in the version stack not just the entire stack.

Content Ratings
With this feature, each document in a list has a number of stars shown alongside it and users can rate the content. Later you can sort the documents by popularity. At first sight, this feature looks a bit like "a gimmick looking for a home" rather than a true enterprise content management feature, but after using it I realized that it is actually quite useful. Assuming that the people you share content with are like-minded, the democratic approach to deciding which document is most appropriate is amazingly accurate.

Document Sets
Document Sets should be something to get excited about, but alas they are not quite everything that they should be in this first iteration. The general concept of a document set is that it allows you to group a set of documents together and treat them as a single object.

They share a single set of metadata, can be versioned as a group, downloaded as a single Zip file and routed through a workflow as a group. The set also includes a Welcome Page that users see when they select the set. This page allows you to add some context to the group of documents. So what’s missing?

* A document can only be in one document set at a time. The concept of related documents is often used to represent a compound document, in this example it would be reasonable to have a common piece of content shared across many the document sets – a standard set of terms and conditions for example. 
* Documents do not have an explicit order within the set. Often the order of the component parts of the document in a collection is critical to whether the end result makes sense. You could order the documents implicitly by keeping the object names ordered alphabetically but that’s a little 1970’s for a product with 2010 in its name. 
* If you perform a search and find a document you have no way of knowing that the document belongs to a document set. From a contextual and compliance perspective this might be undesirable because the document’s inclusion in the document set is almost certainly relevant. 
* The ability to take a snapshot of the specific versions of documents in a document set and protect that snapshot is missing. This model allows you to freeze a copy of the documents as a record but then continue to update the documents in the live set and is often used in records management.

Advanced Content Routing
In SharePoint 2007, content routing was primarily a feature of Record Centers; SharePoint 2010 extends this functionality to all sites. With content routing you establish rules that determine where a document will be routed to based on the metadata of that document.

This feature allows you to create "drop off" areas in your sites; content can be dropped in to these locations and the rules will then file the content away in a predetermined location. Assuming your users tag their objects correctly this feature can save a lot of mistakes.

UI Changes
We are probably all aware of the new ribbon-oriented UI in SharePoint 2010 but there are some other more subtle UI changes hidden away in SharePoint 2010 which directly affect how you will interact with unstructured data.

These changes include Metadata Driven Navigation, advanced targeting, better business data access, RSS feed monitoring, key performance indicators, summary links and searches. Let’s consider the first two of these as they are the most relevant to this article.

Metadata Driven Navigation
If there’s one feature of SharePoint 2010 that encapsulates the approach that Microsoft have taken to large scale ECM deployments it is Metadata Driven Navigation. Being able to store 50 million documents in a single SharePoint list is very impressive but unless your users can find the ‘one in 50 million’ document that they need then you’re not really being successful.

Metadata Driven Navigation allows managed metadata fields to appear in tree-view controls on the left-hand side of the navigation pane. Users can then select values for any field to constrain the results. For example, "show me only final versions of contracts."

Behind the scenes you are really adding constraints to the search predicate and seeing the results in real-time. I’ve heard people refer to this as "zero typing search," which makes sense. You can elect to add these managed fields to the index which seems like it would be a good idea.

So why is this feature so useful? Two primary reasons, firstly it allows you to remove huge swaths of irrelevant documents and hone in on the ones that you need and secondly it is familiar – most online shopping sites use this approach.

Advanced Targeting
Along the theme of getting access to content quickly and easily, SharePoint 2010 introduces the ability to customize page views based on membership of audiences. Within Central Admin you can set up rules which specify who will be a member of specific audiences.

You can then specify which Web Parts, items and links will be visible on a page based on those audiences. This allows you to better target specific content to specific users within your environment.

Offline Support
SharePoint 2010 includes a rich desktop client called SharePoint Workspace, (a replacement for the old Groove product). There are a few ways of looking at this client; its primary use case is to allow you to work offline by caching a copy of documents and data to your local machine.

Workspace can cache an entire site’s content including custom lists, line of business data and even InfoPath forms. If you edit content when you are offline Workspace will push those changes back up to the originating site when you re-connect.

Workspace can also be viewed in a couple of other ways. It gives an alternative view of the SharePoint site’s content even when online; some users might prefer this non-Web interface. It could also be utilized as a solution for accessing your SharePoint sites in environments that have connectivity challenges. The synchronization of content in both directions is asynchronous so you can make changes to documents and then let the systems synchronize those changes in their own sweet time.

Office Web Applications
Microsoft is making a lot of noise around the new web–based versions of Word, PowerPoint, Excel and OneNote. For sure, these new applications fall into the unstructured data camp but right now the best application that Microsoft seems able to come up with is to interact with the Office documents “when at a shared terminal at a conference or at a coffee shop” . I’m sure that there are a multitude of better applications – if you know of a real one, contact me at my blog.

SharePoint List Item Limitation
In SharePoint 2010 you can have up to 50 million items in a SharePoint list. This is a really bad idea; seriously you should probably look for a new system architect when the number of items in a list is six digits long, ever mind eight!

That said, your lists will grow and SharePoint includes some interesting ways of limiting how large a search query can get so you don’t have users taking down the system by entering too broadly bounded queries. You can limit how many objects in an end user’s query results and you can then allow admins and processes to bypass that limit. You can also allow the limit to only be broached within certain times of the week.

Not Creating Unstructured Content in the First Place
Most of us tend to open Word when we want to type something, Excel when we want to calculate something and PowerPoint 10 minutes before we are about to present something. When one looks at many of the Web 2.0 features of SharePoint 2010 you realize that perhaps we should give more thought to our choice of tools.

SharePoint 2010 has broad support for the wiki paradigm; it has native blog management, activity feeds and discussion boards. I wonder how often we should be using these (semi) structured tools instead of just defaulting to creating a monolithic piece of content. Just a thought…

Your Homework
There are a plethora (well quite a few) other content-related features that are not covered in this article. Some of them are worth further investigation. I suggest taking a look at SharePoint 2010’s record management, the new web content management capabilities, all of the Search features (especially ‘refiners’ which allow you to quickly filter result sets), Business Connectivity Services, RSS feed monitoring, key performance indicators, and summary links.

Managing Semi-structured data in SharePoint and what’s new for ECM fans in SharePoint 2010

So, it may seem like I have forgotten how to write but in reality I’ve been writing, just not here! I have two papers complete which I think my usual readers will be interested in. One is for Information Management Magazine (ARMA’s publication) and the other for SharePointPro Connections Magazine. I need to wait until they are published before I can post links to, quote them or discuss them in detail but here’s a brief synopsis of each of them just to get you all excited:

Information Management (ARMA)

As well as publishing IM magazine ARMA represents information management professionals and records managers. They asked me to write an article about how records managers could manage the new wave of social data types – Blogs, Wikis, IM, Activity feeds, polls, discussion threads, etc. Given that SharePoint purports to manage most of these they also asked that the article would have a SharePoint focus.

This one started out well, I defined the data types and the issues at hand. I initially called this type of data ‘semi-structured’ but it turned out that the DB world already had dibs on that phrase so after talking to Microsoft I coined/stole the phrase ‘social data’. Technically what I analyzed was data that was structured at the back-end but had an associated presentation layer that makes it look unstructured to an end user.

It all fell apart when I failed to find anyone with an ‘off the shelf’ solution to the problem. Thankfully I had some great conversations with the SharePoint product team and it turns out that SharePoint 2010 does include some basic capabilities. I stitched that together with some of the custom solutions that I discovered and it all came together rather well.

SharePointPro Connections Magazine

This article is a summary of what’s new in SharePoint 2010 from an ECM perspective – the article highlights the key content management-related features and questions how well they might address enterprise requirements.  I cover most of the new ECM-related features and dig down into some of them where I feel there’s either a misunderstanding wrt what the feature really does or where I think Microsoft missed the mark. IMHO this is not as interesting as the ARMA article which digs into new territory but it is a good primer for SharePoint 2010.

These types of articles frequently get me into trouble because SharePoint fans think that I am bashing SharePoint and my Documentum colleagues think that I am glorifying SharePoint. My solution is to take a truly agnostic view of the factors and try to be honest. I am not known for my tact but this does seem to be working so I’ll stick with it. I’ve also taken to talking to insiders at Microsoft to make sure that what I am saying is at least fair – maybe not palatable but fair.

I’ll post links to the articles as they get published…

Full-blown SharePoint Access from an iPhone 3G

OK, if that Blog entry title doesn't get me 1,000 hits a day then I give up. iPhone 3G and SharePoint both in the title!! The 3G is my newest and mostest favorite toy right now. It is not great as a business device, (no copy/paste, no email search and typing on it takes some getting used to), but as a general productivity device it rocks. It is extremely easy to use, reading emails and web pages is a joy and the free bubble wrap popping application keeps me going during project review meetings.

It got me to thinking that there are some similarities between the iPhone and SharePoint...I really do need to get out more. Here's my thinking...consider the iPhone in comparison to a Blackberry. The iPhone,

• Is amazingly easy to use - in fact you do not really need end user training,
• Includes things that make tasks that bit more fun during the day,
• Is aimed at the needs/desires of the end user,
• Is designed to be pervasive and provide 80% of the functionality that I need 80% of the time,
• Looks like it does everything that a Blackberry does if you read the marketing literature but it isn't quite there,

Now consider the same analogies with SharePoint and a classic ECM solution. SharePoint is intuitive, fun to use and definitely aimed at the end user community. It does provide a lot of the functionality that you need a lot of the time and according to the marketing fluff out there it is a panacea for the enterprise. The comparison falls apart completely when you start to look at the interaction between an iPhone/Blackberry and SharePoint/classic ECM... iPhones and Blackberries don't interact but it is obvious that SharePoint and traditional ECM systems actually work well together!

Oh the title, the image and the reason for this post? I faked the image and but the concept is real; it relates to this article...but truth be told I just wanted to increase my daily hit rate

SharePoint Archiving #3 - EBS vs. RBS...the ultimate grudge match.

 This is part three of a thrilling series of entries related to the aggregation of SharePoint content. It relates back to reference architecture #7, RBS vs. EBS vs. Content Transfer vs. Shortcuts and An Overview of the Potential Solutions. In the previous entry I rambled on ab out the different options available to support the aggregation of data behind a SharePoint deployment however I left you hanging on for more about two recently added technologies. In this entry I’ll go in to more details about these sexy new options, why you might care and the pros and cons of each. I'll do my best to be technically accurate but bear in mind that I am a manager now so just stringing together non-monosyllabic works is difficult enough for me. My main concern is not the actual implementation details; I am more interested in whether either solution might deal with the inherent business problems.

RBS and EBS Review

The Problem...SharePoint stores everything related to an object in SQL Server. This includes the:  

1. Content, (PDFs, PPTs, Zip files, etc.),

2. Metadata, (the object's title, project number, format, etc.)

3. Context, (which site it came from, the folder location, security details, etc.)

#2 and #3 belong in SQL Server because they are represented by structured content. Storing #1 in a database is a travesty of the highest order. I've seen system architects tarred and feathered for doing this. Why? Databases excel at managing lots of ickle bits of data but they suck when it comes to managing large binary objects, (called BLOBs - Binary Large OBjects). Go read more about these issues in the Eight Reference Architectures series. I have seen estimates that suggest that up to 96 petabytes of data will be archived from SharePoint instances over the next 5 years - that's 96 quadrillion bytes of data...not in to a database me thinks!

The solution...Bottom line, you have to get the binary objects out of the database. Not necessarily out of SharePoint but out of SQL Server. In the previous entry I mentioned 5 ways of doing this but did not dig in to options #4 and #5 - RBS and EBS.

RBS and EBS are both pretty new technologies. Given that they have very similar names it is not surprising that people get them confused so here's a primer:

RBS is implemented by SQL Server (only SQL Server 2008 and later); it is nothing to do with SharePoint directly. When you enable RBS, all BLOB streams that SQL Server would normally be compelled to store internally are spewed forth to the file system.

EBS is implemented by MOSS 2007 (available as a hot fix to MOSS 2007 SP1 and later). The EBS provider lives at the very bottom of the SharePoint stack, just above the interface in to SQL Server. Just before the BLOB is passed to SQL server the EBS provider gives your process the opportunity to optionally take ownership of the BLOB.  You give SharePoint a token in exchange so it knows how to get the object back from you at a later date.  

RBS vs. EBS…

There are pros and cons to both approaches and the balance will change over time according to the SharePoint product plans that we know of. Let me spoil the ending for you…I’d recommend EBS today but RBS later as it matures. Here’s the rationale:

Remote BLOB Storage (RBS)

Pros:

• RBS is implemented in SQL Server and is application agnostic. That’s to say, if you turn RBS on then all BLOB objects from any SQL Server-based application will be externalized. If that’s what you want to happen then that’s great but if you need to be able to apply business logic to what is externalized and whence it goes then you are severely restricted.
• It is simple – you turn RBS on and the content is simply stored on to the local file system. If you have some kind of file system virtualization software in place then you can do some basic management tasks but only based on the file system attributes of the object.
• If you want access to the context and metadata of the object then you are going to have to dip in to SQL Server and start hunting down SharePoint based reference information; Microsoft do not recommend this - in fact they do not publicly publish the DB schema for SharePoint so it would be potentially dangerous.
• The current thinking is that RBS might have more longevity than EBS. It is likely that EBS will fade out of the stack over time – obviously this is not 100% certain but likely.

  Cons:

• Getting the content out of SQL Server only solves 5% of the real issues according to 9 of my 10 personalities. Seriously, getting the BLOBs out of SQL Server gives you scalability but it does not deliver any of the IT efficiencies, compliance overlays, or re-purpose/re-use benefits of managing the externalized content.
• Intelligent archiving is the key to getting this right. You need to have the BLOB, the metadata, the context and the ability to manage the object – no less than this. The RBS model only provides the BLOB – no context and no ability to manage the object.
• No business rule mapping…RBS is all or nothing – you get all BLOBs all of the time. EBS is not much better but does support certain rules. For example, in theory you could configure EBS to not externalize content from certain sites or content less than 50KB in size.
• Needs SQL Server 2008 – not a huge deal but a consideration.

  External BLOB Storage (EBS)

Pros:

• EBS is provided by the SharePoint team and although it is lacking in some areas it does understand the context of the BLOB that it exposes. In other words, we do know what the BLOB object is and we can track changes/deletes on the object.
• The architecture allows is to provide an intelligent process for capturing the BLOB and just as importantly for returning the BLOB on demand, (i.e. when you want to view it from SharePoint).
• Because we are interacting directly with the SharePoint processes we can perform more intelligent operations. For example, if the BLOB was deleted (with good reason) from the store then we could cascade that delete back up to SharePoint. Same with changes to the object or its status.
• It does not require SQL Server 2008.

Cons:

• There are a lot of areas where I would improve EBS but for what we are doing at this point in time the only con is that EBS will probably not survive in the long term. For what it is worth, we have worked with Microsoft to ensure that a transition to RBS in the future would be seamless.

The Bottom Line

The fact that Microsoft have provided mechanisms to allow for partners to hook in to the underlying storage capabilities of SharePoint is testament to the fact that Microsoft recognize the value that other companies can add to SharePoint. I am often asked whether Microsoft might not just add all of the capabilities of a classic ECM solution to SharePoint - obviously they could but take it from me, they'd be better off focusing on usability, integrations, information worker productivity efficiencies and nailing the Office integrations - that's their sweet spot. It took us 15 years to build up the suite of ECM functionality that you see today and it was painful!

So what's next?

Not surprisingly we have a set of products that leverage all of the pros of this new architecture and that have been designed to add all of the benefits of classic ECM without taking away anything from the SharePoint user experience. Contact me if you need more information under NDA.

How do I know whether I *need* a traditional ECM system with SharePoint?

I was speaking at an AIIM conference in Chicago yesterday (http://www.aiim.org/ecmseminar/). The topic of my talk was how traditional ECM systems add value to SharePoint. I rambled on about all of the value adds and at the end, someone asked me how they would know whether SharePoint was good enough as-is or whether they needed to supplement it with an ECM system. I thought that this was a great question…

The answer might be, “if you do not know whether you need one then you probably don’t” but the answer is more accurately “if you do not know whether you need one then you probably don’t…yet”. If you are a small company rolling out SharePoint, or a large company doing a limited pilot then you are probably doing a great job managing the process. There may come a time however when you will start to think about long-term archiving of that content or retention management. IT might want to start to implement HSM or to de-duplicate the content. You might want to start to intelligently retain version information, etc…

My answer at the conference was to say that if everything was ticking along nicely today then you are probably OK but what you absolutely should do is think about the future aggregation and intelligent management of those sites today. Here’s the deal…would you allow each department in your organization to roll out their own Microsoft Exchange mail server with its own domain and their own email addresses? Of course not because any future reconciliation of those systems would be a nightmare. Then treat your SharePoint implementations in the same way. I told audience to assume that they would bring the systems together in the future and to that end they should do a few simple things:

· Make sure that there are no maverick deployments of SharePoint in the organization. All SharePoint sites have to be managed by IT – no exceptions.

· Make sure that the user identity management is common across all sites, (probably the most important one.)

· Whenever possible manage the creation of object types especially ensuring that common attributes have the same name and draw their values from a common domain list, (don’t have a project number represented in one site as Proj_# and in another as ProjectNo and certainly don’t make them free text fields!)

· Consider how you would aggregate the sites moving forwards…is there anything else that you might be able to do to make this easier moving forwards?

Here’s a cast iron Chapman guarantee. If you follow the steps above you will reap the rewards even if you never aggregate the content or work with a traditional ECM system. Don’t believe me? Just consider how much easier eDiscovery would be if you followed the steps above? Need to find all of the drivel that Andrew Chapman wrote? If I have the same user identity in all SharePoint instances then that just became a lot easier.

SharePoint as a replacement for Enterprise Content Management (ECM) systems?

I read an interesting (albeit not too surprising) article by Alan Pelz-Sharpe from CMS Watch entitled "Three continents, one SharePoint story" which claims that customers are not looking at SharePoint as a direct replacement for their ECM systems. His global research suggests that in the main customers are delineating between SharePoint as a collaboration tool and SharePoint as a true enterprise content management system. While I agree with Alan in general I have a few random thoughts...

• In the past, when I spoke to customers about SharePoint and Documentum integrations I used to always hear the question - "Why do I need both?". I'd agree that I am almost never asked that question anymore so maybe customers now have a better understanding of the symbiotic relationship between the solutions.
• One thing that bothers me in Alan's assessment is that he is talking to existing ECM customers which skews the results IMHO.
• Existing ECM customers have an investment in their ECM solution of choice - the cost of moving existing content and processes may be a major factor in maintaining both systems. He mentions this in his 'rip and replace' comments.
• Existing ECM customers tend to be medium-large companies - I wonder what the results would be if the same question was asked of smaller companies.
• I think that just because companies are not dumping their ECM systems doesn't mean that SharePoint is not being used for ECM alongside those formal ECM solutions. Many customers with whom I speak are doing uncontrolled, informal, collaborative content management in SharePoint and then want the documents to 'move' over in to the ECM system at a certain point in their lifecycles.

Of course the real point here is that SharePoint may not really provide good enough ECM capabilities today but what's to come? I still struggle to see how SharePoint could support some of the mandatory ECM-related functionality with their current architecture but it would be naive to think that Microsoft would not go after that space in future releases.

If anyone from Microsoft would like to tell us the 5 year strategy for SharePoint I promise that I'll keep it a secret. Just post it here as a comment.

Microsoft SharePoint – Useless, confusing or just misunderstood?

Firstly, apologies for not blogging recently; I’m working on some innovative and groundbreaking stuff at work right now which is really interesting but I’m not in a position to share it just yet.

Firstly, go and read this article…I’ll wait.

http://computerworld.co.nz/news.nsf/mgmt/1D44BF832AD397E9CC257482007F9B7B

I found the article intriguing because I usually find these articles to be either 90% aligned with my thinking or 90% not aligned, (or “wrong” as I like to call them). In this case I found that I disagreed with certain points but completely agreed with others. Then it hit me…the authors missed one point that is critical to understanding SharePoint deployments in enterprises today; SharePoint is as not really a monolithic application or a pizza pie of functionality, it is really a development platform that comes pre-configured. I find that if I talk to the people who manage successful SharePoint deployments in organizations they completely get this but perhaps there is a disconnect between that reality and people’s perception of SharePoint.

If you assume that this premise is correct and go back and read the article you realize that many of the points are moot – in fact most of them are. That said, the author is right to point these issues out as they do exist, it is just that they are more acceptable in a development platform than a standalone application.

There were a few other sections in the article that I wanted to call out specifically.

"Recognise that a really good SharePoint installation is as much organisation as it is technology," says Creese. I could not agree more however let’s be fair to SharePoint…I’ve never met any ECM system that wasn’t 85% planning and 15% technology…or should I say that I have never met a successful ECM implementation that wasn’t 85/15! Make that any enterprise deployment of any kind of software!

He goes on to add "We have been finding that a highly tuned SharePoint installation needs custom coding and third-party add-ons." His former point falls in to the “recognizing that SharePoint is a development platform” but the second point is potentially more interesting to me. I agree completely and think that SharePoint has opened up a huge net new market for people who understand the subtleties that make for a successful enterprise deployment of any part of SharePoint.

Later Craig Roth also from Burton says “governance is a critical factor in determining SharePoint success. That includes combining people, policies and processes to determine who can do what, when and how.” I think that this a real area for improvement in the SharePoint architecture overall. It is not just about governance – it is about a common centrally managed governance model that includes all SharePoint sites and all other relevant systems in the enterprise.

In conclusion I think that the article brings up some great points and some areas that make me feel that maybe we need to see more clarity in how Microsoft message what SharePoint is…not everything to everyone but a specific layer in an enterprise infrastructure with some very specific functionality.