MemorableURL.com

Full-blown SharePoint Access from an iPhone 3G

OK, if that Blog entry title doesn't get me 1,000 hits a day then I give up. iPhone 3G and SharePoint both in the title!! The 3G is my newest and mostest favorite toy right now. It is not great as a business device, (no copy/paste, no email search and typing on it takes some getting used to), but as a general productivity device it rocks. It is extremely easy to use, reading emails and web pages is a joy and the free bubble wrap popping application keeps me going during project review meetings.

It got me to thinking that there are some similarities between the iPhone and SharePoint...I really do need to get out more. Here's my thinking...consider the iPhone in comparison to a Blackberry. The iPhone,

• Is amazingly easy to use - in fact you do not really need end user training,
• Includes things that make tasks that bit more fun during the day,
• Is aimed at the needs/desires of the end user,
• Is designed to be pervasive and provide 80% of the functionality that I need 80% of the time,
• Looks like it does everything that a Blackberry does if you read the marketing literature but it isn't quite there,

Now consider the same analogies with SharePoint and a classic ECM solution. SharePoint is intuitive, fun to use and definitely aimed at the end user community. It does provide a lot of the functionality that you need a lot of the time and according to the marketing fluff out there it is a panacea for the enterprise. The comparison falls apart completely when you start to look at the interaction between an iPhone/Blackberry and SharePoint/classic ECM... iPhones and Blackberries don't interact but it is obvious that SharePoint and traditional ECM systems actually work well together!

Oh the title, the image and the reason for this post? I faked the image and but the concept is real; it relates to this article...but truth be told I just wanted to increase my daily hit rate

SharePoint Archiving #3 - EBS vs. RBS...the ultimate grudge match.

 This is part three of a thrilling series of entries related to the aggregation of SharePoint content. It relates back to reference architecture #7, RBS vs. EBS vs. Content Transfer vs. Shortcuts and An Overview of the Potential Solutions. In the previous entry I rambled on ab out the different options available to support the aggregation of data behind a SharePoint deployment however I left you hanging on for more about two recently added technologies. In this entry I’ll go in to more details about these sexy new options, why you might care and the pros and cons of each. I'll do my best to be technically accurate but bear in mind that I am a manager now so just stringing together non-monosyllabic works is difficult enough for me. My main concern is not the actual implementation details; I am more interested in whether either solution might deal with the inherent business problems.

RBS and EBS Review

The Problem...SharePoint stores everything related to an object in SQL Server. This includes the:  

1. Content, (PDFs, PPTs, Zip files, etc.),

2. Metadata, (the object's title, project number, format, etc.)

3. Context, (which site it came from, the folder location, security details, etc.)

#2 and #3 belong in SQL Server because they are represented by structured content. Storing #1 in a database is a travesty of the highest order. I've seen system architects tarred and feathered for doing this. Why? Databases excel at managing lots of ickle bits of data but they suck when it comes to managing large binary objects, (called BLOBs - Binary Large OBjects). Go read more about these issues in the Eight Reference Architectures series. I have seen estimates that suggest that up to 96 petabytes of data will be archived from SharePoint instances over the next 5 years - that's 96 quadrillion bytes of data...not in to a database me thinks!

The solution...Bottom line, you have to get the binary objects out of the database. Not necessarily out of SharePoint but out of SQL Server. In the previous entry I mentioned 5 ways of doing this but did not dig in to options #4 and #5 - RBS and EBS.

RBS and EBS are both pretty new technologies. Given that they have very similar names it is not surprising that people get them confused so here's a primer:

RBS is implemented by SQL Server (only SQL Server 2008 and later); it is nothing to do with SharePoint directly. When you enable RBS, all BLOB streams that SQL Server would normally be compelled to store internally are spewed forth to the file system.

EBS is implemented by MOSS 2007 (available as a hot fix to MOSS 2007 SP1 and later). The EBS provider lives at the very bottom of the SharePoint stack, just above the interface in to SQL Server. Just before the BLOB is passed to SQL server the EBS provider gives your process the opportunity to optionally take ownership of the BLOB.  You give SharePoint a token in exchange so it knows how to get the object back from you at a later date.  

RBS vs. EBS…

There are pros and cons to both approaches and the balance will change over time according to the SharePoint product plans that we know of. Let me spoil the ending for you…I’d recommend EBS today but RBS later as it matures. Here’s the rationale:

Remote BLOB Storage (RBS)

Pros:

• RBS is implemented in SQL Server and is application agnostic. That’s to say, if you turn RBS on then all BLOB objects from any SQL Server-based application will be externalized. If that’s what you want to happen then that’s great but if you need to be able to apply business logic to what is externalized and whence it goes then you are severely restricted.
• It is simple – you turn RBS on and the content is simply stored on to the local file system. If you have some kind of file system virtualization software in place then you can do some basic management tasks but only based on the file system attributes of the object.
• If you want access to the context and metadata of the object then you are going to have to dip in to SQL Server and start hunting down SharePoint based reference information; Microsoft do not recommend this - in fact they do not publicly publish the DB schema for SharePoint so it would be potentially dangerous.
• The current thinking is that RBS might have more longevity than EBS. It is likely that EBS will fade out of the stack over time – obviously this is not 100% certain but likely.

  Cons:

• Getting the content out of SQL Server only solves 5% of the real issues according to 9 of my 10 personalities. Seriously, getting the BLOBs out of SQL Server gives you scalability but it does not deliver any of the IT efficiencies, compliance overlays, or re-purpose/re-use benefits of managing the externalized content.
• Intelligent archiving is the key to getting this right. You need to have the BLOB, the metadata, the context and the ability to manage the object – no less than this. The RBS model only provides the BLOB – no context and no ability to manage the object.
• No business rule mapping…RBS is all or nothing – you get all BLOBs all of the time. EBS is not much better but does support certain rules. For example, in theory you could configure EBS to not externalize content from certain sites or content less than 50KB in size.
• Needs SQL Server 2008 – not a huge deal but a consideration.

  External BLOB Storage (EBS)

Pros:

• EBS is provided by the SharePoint team and although it is lacking in some areas it does understand the context of the BLOB that it exposes. In other words, we do know what the BLOB object is and we can track changes/deletes on the object.
• The architecture allows is to provide an intelligent process for capturing the BLOB and just as importantly for returning the BLOB on demand, (i.e. when you want to view it from SharePoint).
• Because we are interacting directly with the SharePoint processes we can perform more intelligent operations. For example, if the BLOB was deleted (with good reason) from the store then we could cascade that delete back up to SharePoint. Same with changes to the object or its status.
• It does not require SQL Server 2008.

Cons:

• There are a lot of areas where I would improve EBS but for what we are doing at this point in time the only con is that EBS will probably not survive in the long term. For what it is worth, we have worked with Microsoft to ensure that a transition to RBS in the future would be seamless.

The Bottom Line

The fact that Microsoft have provided mechanisms to allow for partners to hook in to the underlying storage capabilities of SharePoint is testament to the fact that Microsoft recognize the value that other companies can add to SharePoint. I am often asked whether Microsoft might not just add all of the capabilities of a classic ECM solution to SharePoint - obviously they could but take it from me, they'd be better off focusing on usability, integrations, information worker productivity efficiencies and nailing the Office integrations - that's their sweet spot. It took us 15 years to build up the suite of ECM functionality that you see today and it was painful!

So what's next?

Not surprisingly we have a set of products that leverage all of the pros of this new architecture and that have been designed to add all of the benefits of classic ECM without taking away anything from the SharePoint user experience. Contact me if you need more information under NDA.

SharePoint Archiving #2 - An Overview of the Potential Solutions.

This is part two of a thrilling series of entries related to the aggregation of SharePoint content. It relates back to reference architecture #7 and part one in the series.

In the previous entry I rambled on about the “why” behind data aggregation. In this entry I’ll give you a high-level overview of the “how”.

What are my options?

I think that it is fair to say that up until recently, (prior to MOSS 2007 SP1 and SQL Server 2008), you only had three options:

1. Move objects out of SharePoint in to an aggregated storage location when appropriate.
2. Do #1 and leave behind a shortcut so the content is not orphaned in SharePoint.
3. Build your deployment infrastructure around the inherent limitations in SharePoint.

Post MOSS 2007 SP1 and SQL Server 2008 you can add two sexy new options to the list.

1. Use Remote Blob Storage (RBS).
2. Use External Blob Storage (EBS).

I am going to briefly describe all 5 options and then in the next posting I’ll discuss in more detail the epic battle between the forces of good (EBS) and the forces of evil (RBS). (Only kidding SQL team, I am sure that RBS’s mother loves it!)

1. Move objects out of SharePoint in to an aggregated storage location when appropriate.

The premise here is that you would start to move content from SharePoint in to the aggregated store based on some business criteria; for example when an object becomes a formal record. The object goes from being owned by SharePoint to being owned by the aggregated storage device - and no longer accessible from SharePoint.  In some cases this is fine, (for formal records management for example), but it is invasive and typically I would not call it a scalable solution - i.e. it does not really suit all use cases. One other consideration is that you are not really dealing with the underlying challenges that SharePoint's silo-oriented architectural model creates, you are just bypassing them.

Perhaps most importantly, this solution is technology driving the deployment model rather than the business needs driving the deployment. Of course it almost goes without saying that the end users hate it; their precious documents just disappeared from their SharePoint site never to be seen again...hardly the answer they were looking for!

2. Do #1 and leave behind a shortcut so the content is not orphaned in SharePoint.

In this case we move content out of SharePoint just like in #1 above but we leave behind a shortcut/proxy object/stub/pointer/reference object, (that's one thing not five different things - this concept has more aliases than a mafia boss in the witness protection program). The shortcut is a pointer from SharePoint back to the object that was moved in to the aggregated store.

When you write it down, see a vendor led demo, or read the marketing literature this concept sounds "just peachy". Conceptually, it gets the content out of SharePoint but allows users to still access the content from within SharePoint. Wow, what's better than that? Truthfully, the only thing better than that would be a solution that actually worked in reality not just in marketing ga-ga land. (No offence to marketing people, some of my best friends "work" in marketing.)

Shortcuts sound great but I am yet to see any vendor really implement a seamless, working shortcut in SharePoint. SharePoint inherently doesn't support this concept so some actions kinda work but most just don't. SharePoint shortcuts are a bit like DOS/Windows shortcuts, on one level they look like a great idea but on the other 99 levels they suck. If you want to test this then try routing a shortcut through a workflow and see what actually moves through the workflow, (a clue - it is the shortcut object not the document). Try managing links between documents when one of the target documents gets “shortcutted”, (a clue - they break because the object name changes from wibble.doc to something like wibble.doc.aspx). Try updating the object’s metadata from within an Office application when it is represented by a shortcut, etc….

3. Build your deployment infrastructure around the inherent limitations in SharePoint.

More, you want more?

The Oliver Twist approach to SharePoint data management. Just don’t let the SharePoint repository get too full; when it starts to get a bit bloated just crank up a new deployment and start again. Bizarrely enough this is the most common approach that we see today…scary isn’t it?

What's wrong with this approach? Again this is technology driving the deployment model rather than the inherent business need. Also, you are actually making the problem worse because you are creating even more silos.

4. Use Remote Blob Storage (RBS)

The next installment in this series goes in to details surrounding the pros and cons of RBS; I’ll just give you an overview here.

RBS is implemented by SQL Server (only SQL Server 2008 and later); it is nothing to do with SharePoint directly. When you enable RBS, all BLOB streams that SQL Server would normally be compelled to store internally are spewed forth to the file system. There is no selectivity – all BLOBs get up-chucked. Note that because this happens at such a low level, you do not have access to any of the SharePoint-related information about this BLOB. Without accessing the SQL tables directly, you do not know what it is, from whence it came, its security settings in SharePoint, its relationship to other versions, etc.

5. Use External Blob Storage (EBS)

The next installment in this series goes in to details surrounding the pros and cons of EBS; I’ll just give you an overview here.

EBS is implemented by MOSS 2007 (available as a hot fix to MOSS 2007 SP1 and later). The EBS provider lives at the very bottom of the SharePoint stack, just above the interface in to SQL Server. Just before the BLOB is passed to SQL server you have the opportunity to optionally take ownership of the object. The metadata and context related to the object continue in to SQL Server but you get the actual binary object to manage.

Remember, if you elect to take the BLOB then you are also responsible for making sure that you can retrieve it when the EBS provider asks for it, (when a user views the object for example).

RBS or EBS?

These two technologies have their pros and cons…I’ll go in to more details in the next posting…stay tuned.

BTW - If I have missed any options then post a response and I'll add them...

SharePoint Archiving #1 - RBS vs. EBS vs. Content Transfer vs. Shortcuts

If you only read one of my series of posts this lifetime you might want to make it this one. It is certainly not my most coherent or interesting series but it is the one that is likely to save you the most grief over the next few years, (assuming you have something to do with SharePoint deployments).

This was going to be a single post but I got a bit carried away so I've split it up in to the following posts:

1. This one - An Overview of the Issues
2. The next one - An Overview of the Potential Solutions.
3. The one after that - EBS vs. RBS...the ultimate grudge match (because I begrudge having to deal with it.)
4. Maybe a bonus final posting but probably not.

An Overview of the Issues

I have dedicated a lot of posts in the past talking about how and why it makes sense to create a nice symbiotic relationship between your ECM system of choice and SharePoint. I won't ramble on about silos, compliance, long term archiving, over duplication of content, scalability etc. ad nauseam (again) instead I will take a different tack.

Imagine a world where content created in SharePoint was automatically routed to the most appropriate location depending on factors such as values in the object's attributes, where the object is in its lifecycle and/or who created it. Imagine that this was done without in any way affecting the SharePoint end user experience or any applications built on top of SharePoint. Imagine if doing this didn't just reduce risk and costs but it also made your SharePoint deployments more scalable and robust.

Before we continue to dream of such a thing let's talk about one of the fundamental issues with the SharePoint architecture. SharePoint stores everything in SQL Server, not just the structured content  but also the unstructured content, (the actual binaries - Word documents, PDFs, JPGs, etc.). If we could get these binaries out of SQL Server and manage them in a more appropriate way then many of the limitations and concerns around SharePoint are lessened.

Let's start by reviewing why getting content out of SharePoint’s back end is a big deal? You might be asking what's wrong with how SharePoint manages content today; to save you going back over my Blog to read the answers let me give you a high-level overview.

How SharePoint Manages Content Today.

• If you import a 10GB object in to SharePoint and fill out the properties screen the 10GB object and its associated metadata cascade down the SharePoint stack and both end up in SQL Server.
• The attributes get written to a database table and the 10GB file is also stored in the database as a Binary Large OBject (BLOB ).
  • In my ever so humble opinion the only thing good about BLOBs is the cool acronym. When I was a DB admin we were told never to store binaries in the database unless there were lots of tiny winy wittle files - XML for example.
  • In fact, many of the limitations that you hear about SharePoint's scalability actually come from SQL Server trying to store the BLOBs not specifically from SharePoint or IIS.
• If you import that same 10GB file 100 times then it will be stored 100 times, indexed 100 times, potentially discovered 100 times per case, backed-up 100 times per backup, you get the idea. Turn on SP versioning and you can add a zero to those numbers BTW.
• Content stored in an active database-based application is difficult to archive for long term retention. In other words, if you need to keep your content for 25 years then you'll have to keep your SharePoint sites and SQL databases running for 25 years...good luck with that one.
• SQL Server databases tend to run on very expensive hardware, (I'm paid by EMC so you wonder why I'd care but maybe I'm just bursting full of altruism today). There's an IT concept called Hierarchical Storage Management (HSM) that allows you to move things on to less expensive and typically less available devices based on predicted demand. Basically, throw the stuff on a cheap slow device if no one is likely to want to use it much.

In summary, storing the BLOBs in SQL Server creates issues related to scalability and related to the creation of silos.

How bad is using BLOBs really?

That really depends on who you are and what SharePoint is being used for. For small, non-regulated deployments you may not care, however if you want to use SharePoint as part of your enterprise infrastructure then it potentially really sucks. For example, there's a good chance that you will be forced to structure your site and farm topology based on SQL Server capacity rather than based on the business need. Technology driving deployments is never a good idea...in fact it always bites you in the end (the rear end usually).

What would avoiding Database BLOBs give me really?

Oh, I am so glad I asked, (I am a Gemini so this split personality thing is perfectly normal according to my team of psychiatrists). I use the term “Data Aggregation” in my architectural postings to describe the concept of storing all of the binary objects in a single location. Here's the picture to hold in your mind: binary objects that are being managed by SharePoint would be stored in a single centralized system; not all of them all of the time but whenever it makes sense. Along with the binary object we also take a "convenience copy" of some of the object's metadata and any other contextual information of interest, (the folder it came from for example). This metadata and context data is captured simply to allow us to be able to work intelligently with the object.

So what would this aggregated view give me? Let's break it down in to three key areas:

1. Operational Efficiencies
   • Use the metadata values to intelligently to perform HSM on the objects – i.e. move content to different storage devices based on a set of rules. This reduces hardware costs considerably.
   • While you are at it why not de-duplicate the content, (think of the storage and backup savings there). This de-duplication is not just within a single site, you could de-dupe across all SharePoint sites in the entire company. Bear in mind that if you have versioning switched on in SharePoint it tends to create a huge number of identical copies of documents…more than you might expect.
   • Don't stop there; you can also more efficiently deploy your SQL Servers because they will scale up the wazoo if they only contain the structured data. ("up the wazoo" is a DBA technical term.)
   • Backup and recovery…well the jury is out on this one. Depending on where you aggregate your content to and how you do your backups (hot vs. offline) this could either be hugely efficient or a bit of a nightmare. We are working on that one - if you have any ideas contact me directly for a chat.
   • You will certainly increase the scalability of your deployments by removing the payload from SQL server; this alone might justify the effort to aggregate your data,
   • Many companies do not manage all of their SharePoint deployments from within the data center simply because they do not have the capacity to service that many separate systems. This aggregation approach means that IT could allow departmental rollouts of SharePoint but they would mandate that the deployments utilize the aggregated store so IT has responsibility for the actual content.
2. Governance, Risk and Compliance (GRC)
   • Once you have all, or a subset, of your content in a central repository you can start to apply good governance controls to it. If it is an ECM system you can start applying retention controls, you can apply digital rights management to objects and have more robust data protection/audit controls.
   • Leveraging the HSM feature you can apply different levels of control to different categories of content. For example, some content might just be stored on a secure file system, other content might be dumped in to your ECM system and some might go to a CAS device.
   • Probably the biggest compliance advantage is the ability to centralize the application and management of your controls. Given that all of your corporate assets are now in one place you can apply a common set of controls to them – one file plan, one set of retention schedules and one disposition process.
3. Business Gains
   • What about your end users? To be fair, they’d probably love this system simply because it is non-invasive but we can throw them another bone or two just to be kind.
   • eDiscovery is a case in point – obviously having everything in one place, de-duplicated and indexed is a boon for the legal-eagles but there is another bonus feature…as I mention in the first list, right now there is a plethora of content lurking in the hidden department-deployed SharePoint systems; aggregation means that you can ensure that you have visibility in to all of this content.
   • Long term archiving is another interesting use-case; this could be compliance driven or just good business practice. If your content resides in SharePoint then in order to have access to the content in 20 years time you will have to still have SharePoint running. This is neither attractive nor realistic! However, if the key business documents have been archived out with their metadata then you have many more options such as leaving them in your existing long term repository, exporting them to a file system with an XML tag file, storing them on a CAS device, etc.

So, hopefully you now have a feel for why you might want to take a look at aggregating your content in to one location. In the next post I’ll discuss the different approaches that are available and then in the third post I’ll focus on two new capabilities that Microsoft have made available – EBS and RBS.

Content Management Interoperability Services (CMIS) News

I am catching up on my Blog entries after the run up to Momentum Europe and the blur that was a trip to Prague...here's another one that I should have posted 3 weeks ago!

I usually avoid blatant EMC Product information regurgitation but actually this one is worthy of a brief posting. We have made an early access release of our CMIS integration available for download. It works with Documentum and can be downloaded from the following site: https://community.emc.com/community/labs/cmis

Here's the blurb that arrived wrapped in a $20.00 note yesterday, (I might be cheap but my work is mediocre so that balances it out nicely.)

EMC Documentum is providing an Early Access Release of its CMIS implementation to enable customers and developers to familiarize themselves with the proposed standard. This will provide both developers and customers the information they need to jumpstart development or ensure that they include commitment to CMIS compatibility in future purchases and acquisitions.

Go crazy...