MemorableURL.com

SharePoint Archiving - What, Why & Where?

I'm down in sunny Orlando for the Microsoft TechEd Conference this week which is odd because I am nether technical nor particularly educated; maybe I'll return a different person. One of the things that I'm going to be validating with customers, partners and the Microsoft team is the general concept of "SharePoint Archiving."

It would seem that one man's archive is another man's backup/journal/file system. This has not always been the case but these days we seem to overload the term 'archiving' way too much IMHO. In fact, the definition of an archive seems to be multidimensional; based on why you archive, how you archive or even where you archive from - I'd proffer that there is no right or wrong definition, just different ones. At the end of this piece I'll suggest specific terms to describe each 'flavor' of archive that might help clarify the different approaches.

There are three main areas that I am considering right now:

1. What do we mean by an archive?
2. Why bother archiving at all?
3. At what level of the SharePoint architecture would you perform the archiving?

Let's consider each of these in turn - if you are at TechEd and you have an opinion that you'd like to share then head over to the EMC booth and they should be able to get hold of me.

What?

This seems like a redundant question - "What is an archive?" seems like a question that contains the answer... let me give you a few different definitions that I've heard bandied around and you'll see how broad a range of definitions there are:

• Some vendors have solutions that copy an entire document library or site to an 'archive' .
• The ability to declare a piece of content in SharePoint as a record can be termed 'archiving'.
• Allegedly, Microsoft call their remote blob storage capabilities 'archiving', (RBS and associated new functionality support the ability to store content on a file system instead of the SQL Server database.).
• If you are the DB admin then backing up the underlying SQL Server database is archiving.

I think that the confusion and hence the clarification actually comes from the next question - "Why?"

Why?

Your personal definition of archiving is going to be driven by the rationale for archiving in the first place. Is it driven by compliance, is it a way to increase performance, does it provide operational backups, does it reduce scalability constraints or is it a means to re-purpose the content? Also consider whether the archive is active or inactive - do you need to archive the content and then continue to access it from SharePoint or will the archive only be used offline?

I think that it is fair to say that until recently an archive was something that took your content offline. Consider the original archives that we had before electronic document management systems appeared - they were physical vaults where your 'archived content' was shipped off to in a box. We continue to use the term archive when the reality of what an archive looks like have changed dramatically.

Where?

This is an interesting question that forms another dimension of the definition of archiving although I recognize that this is the technology solution wagging the business problem, (there's a metaphorical mix that you don't hear too often!)

Will you archive from a very low level by grabbing content from the file system or directly from SQL Server? Will you archive transparently from SharePoint so that the end user does not even know what the archive happened? Will the archive process be something that the end user invokes intentionally (e.g. by declaring an object as being a record).

The 'where' question will determine the number of objects that you will archive and what you can do to the archived content. Archiving content at a low level without SharePoint knowing that you've done it makes it harder to apply extra constraints to the content; for example, if you archive something out of SQL Server and make it immutable and then try to delete it from SharePoint what will happen? If you archive content as a manual process at a higher level in the stack then the chance is greater that you'll be able to interact with the archived content intelligently.

Who Cares?

I think that we all should; we cannot just keep throwing content into SharePoint without an 'exit strategy' for the content. If you don't know what is going to happen to the content in the future then you'd better start saving up for all of the nice EMC hardware that you'll need!

Chapman's Definitions:

As promised, here are my definitions for the different types of archiving...

• Archive: A copy of content and/or metadata that is stored offline for compliance purposes. This includes formal compliance requirements like records management and also informal business-driven/best practice compliance such as keeping copies of documents for historical reasons.
• Backup: Differs from an archive because the sole purpose of the copy is to allow the content to be recovered for operational reasons.
• Journal: Differs from an archive because the content is moved from SharePoint in to the journal provider and remains online. Not all content needs to be moved to the journal - maybe just content that is over a specific size or of a certain type.
• Centralized repository: Similar to a journal but the content is consolidated in to one location, (a journal solution could legitimately have one journal location per SharePoint site.) This approach allows you to have an aggregated view of content across sites and to then manage the content en masse.

I'm sure that this list is not comprehensive; I'd love to add more definitions to it and also tighten up the scope of each one. If you have feedback I'll incorporate it.

Questions from " And the winner is... "

KDiddy responded to my EMC World write-up with a list of questions that arose from attending my session, (it is never good when people leave your sessions with more questions that when they arrived…if I was thought provoking then it would be great but I think I’m just confusing!)

Rather than burying the response deep in the comments section I’ve posted the answers as a new entry.

WHO ARE THESE CUSTOMERS THAT YOU TALK TO?

Well, there’s Fred, Eleanor, Bob and Arthur – you’d love them, especially when we get them drunk. Seriously, we also try very hard to talk to a real cross section of sober customers – and not just existing EMC customers, (that is especially important for a project like this.) I assume you are not asking me to name names but in general we talk to:

• Non-EMC customers, especially ones who are very Microsoft-centric,
• EMC ECM customers who are typically approaching us with very specific requirements,
• We run customer focus groups where we validate ideas and dig deeper in to the needs,
• We also have a lot of indirect and anecdotal contact with customers through talking to the field and even reading Blogs etc.

My only admission of impropriety would be that the customers we talk to are typically from medium, large and vast companies. I don’t spend too much time with smaller companies.

WHAT IS THEIR MOTIVATION? IS IT SELF-PRESERVATION?

• Not sure exactly what you mean – are they trying to protect their ECM empires or trying to stay out of jail for non-compliance. Either way I’d say that there is an element of both of these but the positive spin is probably to say that many of our customers understand the value of ECM so they are protecting ECM in their organization for all of the right reasons. In fact, one of our biggest challenges is helping people to understand “why ECM”…not just to protect our business but to help them manage operations, risk and compliance.

MORE INTERESTINGLY IS HOW/WHY DID SHAREPOINT BECOME SO PERVASIVE IN THESE ORGANIZATIONS TO THE POINT THAT IT IS NOW AN ISSUE FOR THESE CUSTOMERS TO SOLVE?

• Great question… IMHO Microsoft did a great job of creating a solution in a space what had almost no friction. The layer in which SharePoint resides is so multifaceted – it can be viewed as an über-file system, an extremely rich development platform or a layer under Office that provides basic productivity tools. Previously there was no easy way of extending Microsoft Office in to an organization’s processes. Oh, and people believe that it comes for free…it doesn’t but even the illusion of a bargain is a great sales tool.

DOES THE SPREAD OF SHAREPOINT, REGARDLESS OF WHETHER YOU BELIEVE THE 100M USER STORY OR NOT, CONFIRM THAT THE INCUMBENT ECM SOLUTIONS WAS NOT WHAT MAJORITY OF FOLKS IN THE ORGANIZATION WANTED?

• Let me play devil’s advocate…you make the assumption that SharePoint is being used for ECM, and that it is able to supply real *Enterprise* level content management capabilities. I’d argue that is does not really do that; it provides basic library services which is a long way from ECM. I have three thoughts, (really since birth…only three of them):
• SharePoint’ successful is not based on its ability to provide an ECM capabilities; that’s just a small part of why people use SharePoint. Its success is based on a much bigger picture.
• In many companies SharePoint is not viewed as being a competitive technology to ECM….really, it is not. It is seen as being a cooperative technology; people want to build solutions on the SharePoint platform, use it for collaboration, as a portal, etc.
• Yes, SharePoint provides some critical things that the legacy ECM solutions have failed to provide…ease of use from the Microsoft Office tools being the most obvious.

IF YOU DISCOUNT MSFT MARKETING MACHINE AND THE FACT THAT A VERSION OF SHAREPOINT IS AVAILABLE FOR FREE; DID SHAREPOINT DELIVER TO THE MASSES THE ECM SOLUTION THEY DESIRED?

• For some customers is absolutely does – no doubt about it. It is not “real ECM” by my definition but it certainly provides the basic library services that some companies need. If you don’t need a sledgehammer to crack a nut then don’t use one. Your nuts will get crushed and nobody wants that.
• The real question is…”if Millicent’s Software Co. had written SharePoint and it was $200 per seat would it be a success?” I think that answer is probably yes but it would have been crushed by the marketing machines from the major ECM vendors. The general concept that SharePoint delivers is very attractive, I just think that it is oversold in many cases.

BY THE WAY, I COMPLETELY AGREE WITH YOU THAT CREATING SHORTCUTS, WEBPARTS AND FRAMEWORKS ARE SIMPLY TECHNOLOGIES THAT HOPEFULLY WILL BE USED TO SOLVE THE UNDERLYING PROBLEM THAT CUSTOMERS FACE.

And the winner is...

Well let's face it, it wasn't me..."post a Blog entry every day". I didn't exactly pull that one off but here are some random thoughts that I do have time to put to paper given that I have four hours to kill in Las Vegas airport. Just in case you care, I don't gamble or drink and I hate people so Las Vegas is about the last place on earth I'd like to be. It is a good job that the good Lord invented Minnesota otherwise I'd be in the actual last place that I want to be

For those of you who don't know me you'll be surprised to hear that I have an eidetic memory which means that I remember absolutely everything...unfortunately I also have A.D.D. which means that I only remember the things that I am thinking about rather than the things I was supposed to be remembering! Anyway, back on track, (for now), I ran through my memory banks and processed everything that was said to me during the conference and here are some interesting facts:

• 73% of the conversations started with either:
  • Are you the poor sap that got landed with the SharePoint integrations?
  • We need you to make SharePoint and Documentum work together...can you sort that out for us this week?
  • If you don't stop doing that I'll call security
• The most common words/expression used in the conversations were:
  • Portal
  • Ease of use
  • I will use this Taser if you don't leave me alone
  • Maverick, (not me, the deployment of SharePoint)
  • One client
  • Search

Focusing on the common words and expressions...there really was a pattern and a few things were validated in my mind. In no particular order:

• People know that they need SharePoint to work with their ECM system but they don't know exactly what that means.
• We joke about the fact that customers call us up and say "I need you to make SharePoint and Documentum work together", we tell them that we will need to sit down with them and better understand their requirements and they tell us, "I just told you them."
• Our customers are usually painfully clear about their requirements; why would they be acting without their normal clarity of thought? My theory is that they don't actually understand what SharePoint is being used for so they cannot clearly define the underlying ECM requirements. If they don't truly understand the role the SharePoint plays in their company then why would be expected to know how it interacts with something like an ECM system.
• The solutions that we all talk about are driven by the available technology...this seems like a good thing but it stifles innovation. We talk about creating Web parts, moving content from SharePoint to Documentum or creating shortcuts as if they were solutions - they are not, they are technologies. Erin, (The SharePoint Product Manager), and I spent some time discussing this after one of the product validation sessions and came to some really interesting conclusions...in fact I'd say we came up with a real humdinger of an idea, (wow, humdinger did not trigger the spell checker). I'd love to share it with you but it was too good...if you are a customer then you'll hear about it over time, I promise.
• Here is an extremely crude survey - from all the people that I spoke to at the conference less than half of them believe that IT are backing up their SharePoint sites. This is probably one of the most startling facts that I have ever published in my Blog. I'm throwing you a bone by burying it here to give you time to sort it out because in the near future I am going to poll my readers and see whether my estimate is close and then Blog about it. There were two overriding facts that I believe contribute to this:
  • IT know that the sites existed but do not have the capacity to back them all up, (or more correctly they did not have the capacity to back up the ones that they did know about.)
  • They did not know where the sites were and because of reason 1 they were unwilling to go looking for the sites...out of "site", out of mind...get it? (Maybe I should drink...)
• I think I am funny...but I'm not
  • That previous pun.
  • I had a slide that announced that I was about to reveal the product roadmap for the new generation of our SharePoint products. The next slide was an image of the "Blue Screen of Death" complete with animated flashing cursor. Most of the people in the room thought that this one was funny, (except the guys in the Microsoft shirts in the front row - sorry).
  • I did my presentation with someone from marketing, (I'll call him Martin Davids to retain his anonymity). "Martin" is from Canada so our Q&A slide had images of some pool cues and one of the Canadian Mounties - Cue and 'A', (the Canadians say 'A' at the end of every sentence-A.)
• If you put the word "SharePoint" in your session title you'll fill the room...even if it is Martin Davids and I speaking.
• There'll be a non-refundable cover charge and 8 drink minimum next year to cover costs.
• On a number of occasions customers compared the SharePoint deployment model to Microsoft Access rather than Lotus Notes which surprised me. They said that when Access came out they found that departments were deploying their own solutions based on Access without IT/Legal being aware. I'm not sure why this is but I suspect that some of them simply never used Notes because they were Microsoft shops so maybe Access was the closest comparison.
• SharePoint as a development platform came up very infrequently but that actually validates to me that the SharePoint community view SharePoint as a platform but for some reason, (the Microsoft marketing machine perhaps), the rest of the world see it as being a solution/application/panacea.

It was a great conference, absolutely exhausting, but really good. The compliance team did an extraordinary job as always and the SharePoint team really took advantage of the opportunity to spread the word and validate our plans. I'd like to thank all of the customers, partners and fellow Bloggers who took the time to chat...Laurence, thanks for not reporting what we really said over lunch, Marko, sorry we didn't touch base - I swear I really was on the phone each time I saw you, I wasn't faking it!

In closing, I'll like to directly quote a customer who was talking about his internal challenges with SharePoint, "...they think it is the Holy Grail but in reality it is just a collection of bits..." 

EMC World Approaching...rapidly...in fact it is here

The run up to EMC World is always hectic but I learned something new this year...when you are involved with a phenomena like SharePoint, the weeks before the conference are totally out of control. Anyway, here we go...wish me luck.

I'm going to try to Blog something each day because there will be lots to discuss. Watch this space...

Sometimes it's not what you say, it's how you say it...

When I have time to spare I love to spend time browsing other people’s Blogs; one of my usual browsing locations is the Big Men On Content Blog. Earlier this month they posted an interesting article which discusses a CMS Watch article about SharePoint vs. Exchange Public Folders. It wasn’t the actual content of Lee’s Blog entry that caught my eye so much as the way that he phrases his thoughts; he is about as subtle as I am, (for which he earns my admiration.) Here are two examples taken from his entry:

• “...Sharepoint, when all the marketing and hoopla is stripped away isn’t ECM anyway. It’s really just Microsoft Portal with a tightly coupled workgroup functions - some of which deal with content...”
• “[Microsoft] wrote their own portlet spec and called them webparts. (arguably the worse name in technology - might as well have called them naughty bits...”

Thanks for continuing to entertain Lee!

Reference Architecture 8: Synchronized, Intelligent, 2-way Shortcutting

Read more about the eight reference architectures.

In this model the ECM system stores the only copy of the actual payload of an object, (the document, spreadsheet, image, etc.), however the object can be managed, consumed and deleted from within SharePoint.

A shortcut is created in SharePoint, the shortcut points to the object in the ECM system and mirrors all of the appropriate object metadata. Assume that you have a Word document called "widget_spec.doc" being stored in the ECM system. The document was created by me on 04/20/2008, I have full access to it and the rest of the world has read-only access. A shortcut would be created in SharePoint that has the appropriate access controls, the object name, title, author, creation date and also the location of the real file in the ECM system, (but not the actual file).

Any actions applied to the shortcut object are redirected to the ECM copy when appropriate. If you click on the shortcut and open the document that action is redirected to the ECM system which grabs the real document and streams it back to SharePoint. The user doesn't know, (or care),that the file came from the ECM system.

Both the SharePoint object and the ECM object are protected by event triggers so that changes to the properties, location, etc. can be synchronized and when either is deleted the other will also be disposed of. This is important because you need to allow the object to be managed from either within the ECM system or from within SharePoint without the shortcut and the original object getting out of sync.

Note that this RefArch works for content that already exists in the ECM system, is being created by a 3rd party process or is "published" from SharePoint. What does this mean? It means that you could move content from SharePoint in to the ECM system and leave a shortcut behind, (kinda like HSM for SharePoint isn't it), you could ingest content in to the ECM system from any source and in real time expose that content in to multiple SharePoint document libraries or you could take existing content in your ECM systems and make it available to one or more SharePoint sites. Attractive proposition isn't it? We call this architecture 'projection'.

I am not going to delve any deeper in to the technology behind this solution for similar reasons to reference architecture #6 but needless to say, this is a very interesting area for ECM vendors right now.

The end...

That's it for the reference architectures for now. Committing these to Blog has been a great therapeutic exercise...thanks for sticking with it!

ECM delivered using SaaS

As I mentioned previously, we had an internal writer's summit a few weeks ago. One of the recurring topics that came up related to how to deliver ECM-related functionality in the brave new world. I'm busy enough trying to keep up with SharePoint and Compliance to dive head first in to the wonderful world of SaaS but I wanted to share some of the specific items that I thought were interesting.

Normally I'd pretend that these ideas were my own but this might be a career limiting move given that the ideas were stolen from people like Howard Shao (founder of Documentum) and Mark Lewis (runs the entire Content Management and Archiving division at EMC)!

SaaS Should Be Delivered Like Electricity

This gem from Howard was memorable primarily because of the simile that he used. Currently, if you are using a SaaS application and you need more capacity, access to extra functionality, etc. you have to call your service provider and make a formal request and they provision those changes - this might typically take days to happen.

Howard's argument is that these services should be available to you completely on demand. The current state of affairs is synonymous with buying a new appliance bringing it home and then having to call PSE&G and asking them to provide another 65Watts of capacity for your house.

Later in the day he added a great addition to the simile. He mentioned that with a SaaS model you may need to have a local cache of content or even some functionality either to supplement performance or for access if the SaaS service is temporarily lost. He compared this as having a generator at your home so when either the grid goes down or when a brown-out hits you have access to extra power locally

Howard's observations are always astute and in this case I thought that this really brought home one of the challenges of abstracting the service from the customer - the customers become ignorant and ambivalent to the operational aspects of the system. Their expection is that the system will be available all of the time, will be as efficient and responsive as every other system in the organization, integrate with everything seamlessly...and they want it for free!

ECM's Unique Demands on a SaaS Implementation

Howard very succinctly summarized, (that's not a pleonasm BTW, you can un-succinctly summarize something), the challenges that ECM brings to a SaaS deployment model. I thought this was worth 'memorializing' so here we go:

1. ECM causes a substantially greater demand on SaaS storage requirements and on the capacity of the network. ECM payloads are typically multi-megabytes in size and users expect content to arrive within seconds.
2. Customers may insist that some types of content are stored on-premise. Customers may feel that very high value intellectual capital should not be stored off-premise. Personally, I think that this is unwise and I think that the concern will change over time - let's face it, if you have a decent SaaS provider then they should be able to protect your content better than you would ever be able to. That said, there are sometimes country-specific compliance requirements that may demand that some content is stored in a certain geographic location.
3. In order to provide adequate levels of security, compliance and performance, ECM systems will need to manage content that is on-premise and off-premise. This will have to occur in a way that is transparent to the end user.
4. The SaaS solutions will need to support tiered storage devices and repositories. Again, this needs to be done in a transparent way.

Hierarchical Storage Management meets SaaS

The last two items above both point to the fact that the SaaS systems will have to be able to abstract from the end user the actual location of the content. Typically we see this type of functionality implemented internally by customers - if you use a decent ECM system it will be able to move content between storage devices based on the value of content or how frequently that content is accessed. Users typically have no idea where the content is stored, they just access it. This concept of HSM is typically just used to move content between different types of local disk storage but imagine if the HSM system was running at your SaaS provider's off-premise location but one of the HSM target location was your local system.

If one of the HSM target locations was your local system then we could set up rules that moved specific content types to your local system if the content was being used very frequently, was highly dynamic or you had a regulation that determined that the content had to be stored in a specific geographic location.

Does TiVo Represent the Ideal ECM/SaaS Architecture?

Someone mentioned TiVo as an example of this type of concept. Many of the broadcast TV companies use ECM solutions to manipulate, manage and store programming; they run all of the systems and provide you with 100 channels of garbage 24/7. However, if you want to keep a copy of some of their programming locally you can manually capture something to your local cache in real time or you can even set rules to cache content to the TiVo box.

Conclusions

I don't really have any conclusions other than to say that SaaS-based ECM is going to happen whether we like it or not and thank goodness that companies like EMC have people a lot more intelligent than I working out how to pull it all together!!

Eight Reference Architecture Organizer

Someone asked me if it was possible to have a single page with links to just the "Integrating SharePoint with Traditional ECM Systems - Eight reference architectures" articles. I think that they were politely saying that they had no interest in my other articles...so this is it, alternatively, you can click here to filter by the relevant category. I'll try to make it "stick" to the top of the Blog.

Reference Architecture 1: Keep Systems Separate, Restrict Usage.

In this architecture the publishing of content between SharePoint and the Enterprise Content Management system is performed manually by the end user. This is not an approach that I’d advocate but it is included for completeness and because it is a method frequently used today simply out of necessity.

Reference Architecture 2: Loosely Coupled Solution

This is probably the key architecture behind any real unification solution today. In this architecture, content is moved between systems depending on its value to the organization. For example, work in progress might live in SharePoint and formal records might live in your ECM system. The move between systems is performed by a process and can be invoked manually or triggered by a specific event.

Reference Architecture 3: Use SharePoint as a Portal Container

In this commonly seen architecture, SharePoint hosts Web Parts that point at your SharePoint repository or at your Enterprise Content Management system. For example, one part of the screen’s real estate is displaying content from SharePoint while another part is displaying your ECM system’s Inbox. This is stretching the definition of “unification” IMHO but for reasons that will become clear later I think that this approach is often essential to make up a complete solution.

Reference Architecture 4: Passive Unification in Web Part

In reference architecture 3 the unification occurs within the SharePoint portal container. In this architecture the unification occurs within a specific Web Part; a single Web Part displaying content from a variety of disparate systems. We will look at this unification model from a passive perspective – only allowing the user to view or export the content.

Reference Architecture 5: Active Unification in Web Part

Building on reference architecture 4, we will see what happens if you attempt to add more complex operations to the affray; supporting operations like checkout, managing versions, connecting objects to a workflow, etc. The addition of these more complex operations increases the complexity exponentially.

Reference Architecture 6: Passive Back-end Aggregation

The first five architectures are all focused on unification. This penultimate example creates a passive aggregation model where we use tools to create a virtual aggregation across the SharePoint and traditional ECM environments. We can then monitor the aggregated environment and take actions as required. For example, we might compare the access rights on a piece of content that lives in 10 different document libraries and report on access inconsistencies.

Reference Architecture 7: Active Back-end Aggregation

The Holy Grail or Pandora’s Box? Real aggregation of the SharePoint and traditional ECM silos allows us to perform many operations that would be very difficult or even impossible otherwise. Having all objects from all environments addressable within a single space and supporting operations on those objects allows us to really address some of the current limitations in a scalable & secure way – think “single instance storage” for example.

Reference Architecture 8: Synchronized, Intelligent, 2-way Shortcutting

In this model the ECM system stores the only copy of the actual payload of an object, (the document, spreadsheet, image, etc.). A shortcut/stub/proxy/pointer is created in SharePoint; this points to the object in the ECM system and mirrors all of the appropriate object metadata. Actions applied to the pointer object are redirected to the ECM copy when appropriate. Both the SharePoint object and the ECM object are protected by event triggers so that changes to the properties, location, etc. can be synchronized and when either is deleted the other will also be disposed of.

Note that this RefArch works for content that already exists in the ECM system, is being created by a 3rd party process or is "published" from SharePoint.

Unification at the Middleware Layer

You may have noticed that the integrations above all manifest themselves in the client or server layer however the majority of SharePoint’s strength lies in the “middleware” layer. I am not ready to discuss middleware-level integrations yet but I expect to be able to add some more meat in to that layer of the sandwich by the time the book is ready to go to press…you may have to part with some cash to learn those bits!

Reference Architecture 6: Passive Back-end Aggregation

Read more about the eight reference architectures.

Let me preface this entry by saying that this reference architecture is similar to reference architecture #1 insomuch as it is not something that I recommend or endorse but it is something that I see in use fairly frequently today; in fact many of the solutions recommended by Microsoft fall in to this category. Best case, feel free to use this approach while you wait for more appropriate solutions to come along.

In the previous architectures we were trying to create a unified model so that an end user could simultaneously see content that resides in the SharePoint systems and also the traditional ECM systems. In this, (and the next), architecture we are going to directly deal with the SharePoint "silo sprawl" in the data center. I call this approach 'aggregation' not unification. The advantage of architectures 6 & 7 is that they are completely transparent to the end user - that's also their failing; these architectures are so transparent that you cannot use them to expose the advanced capabilities of the underlying ECM solutions to the end users. FYI - I am working on a cocktail of solutions that give you the best of both of these worlds but I'll probably keep that under wraps until I have it ironed out a lot more.

The general premise here is to leave each instance of SharePoint running independently as individual disparate systems but to create a virtual aggregation of the content and potentially the metadata behind the scenes. This approach does not actually change any part of SharePoint, the reference architecture's processes and technologies simply monitor the SharePoint implementations and create an aggregated view of the content stored across all libraries. This aggregated view can then be used to make security decisions, perform risk analysis, monitor file usage, etc.

For example, one could create a master index of all content in all SharePoint instances inside of the organization. This master index would contain the basic information about the contents of all of the document libraries along with the security access settings and a hash value of each of the pieces of content. You could then analyze this information and look for instances where a file’s security is not consistent. Let's consider a use case: assume that the company’s year-end financial results are stored in the finance department’s SharePoint document library with very restrictive security in the weeks before the official financial results are announced. If the passive unification process looked in the master repository index and noticed that this same file was also available on one of the finance department's collaborative SharePoint sites with no security then you might want to do something about that! Note that it would know that the two files were the same file even if the file name had been changed because the hash value would uniquely identify the file.

There are plenty of other use-cases for what to do with this data once you have access - monitoring data duplication, managing disposition, etc. but is this really an aggregated  back-end? It isn't really because nothing is actually aggregated in reality but it can be useful because it creates a virtual unified view of the data and once again it is a very non-invasive, low impact approach to the solution.

One issue to consider with this approach is scalability. If you have a billion data objects in your SharePoint systems then your enterprise-wide content index would have a billion entries in it. The other consideration is how to keep this index synchronized as content is added, altered and especially removed from the systems. Those types of operations are very CPU-heavy. 

Talk to the market leaders in database, search and indexing technology for advice on these issues. A billion objects synchronized across an enterprise is theoretically within the abilities of most enterprise database systems however I have talked to customers who generate over 1.5 billion records a day, (yes, that's the right number), these are transactional records currently ingested in to specialized storage but they could easily be stored as XML output files. Assume that they generate 1.5 billion records each working day they will end up with roughly 390 billion objects a year! Scalability becomes an issue very quickly with this kind of ingestion.

Again, I see this as being an architecture that is included for completeness and because it contains some concepts that are useful elsewhere, (think eDiscovery for example). Also, it can be "done on the cheap"; you can achieve passive virtual aggregation using the tools that you own today. A more useful reference architecture is to create some form of real unification at the back end...see the next reference architecture for more on that thrilling concept.

SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution.

Read more about the eight reference architectures. 

In the second of the eight architectures I take the general premise from the previous example and extend it. The content still lives natively in SharePoint while you are collaborating on it and is then "migrated" in to the traditional ECM system at a predetermined point in time.

As a matter of interest according to recent research, for most customers this movement of content from SharePoint to an ECM systems occurs because of one of the following reasons. The document:

• Becomes a formal business record.
• Is being collected as part of a legal discovery process.
• Needs to have extra protection applied, the application of digital rights management for example.
• Is being archived for operational reasons, e.g. the document library is no longer operational and can be archived in its entirety.
• Should be made available for re-use and re-purposing; perhaps being included in a formal structured business process.
• Will be published to web site or an external multi-channel device.

In this reference architecture the migration is done using an automated process which “publishes” the content from SharePoint in to the traditional ECM system. Fundamentally, the automated process is simply extracting the content from SharePoint, caching it locally and then uploading it in to your ECM system but the fact that the migration is controlled by a process not a person allows you to more easily perform some auxiliary tasks. For example, when the content is moved across from SharePoint it is possible to bundle up some of the object’s history, metadata and related content and to move that across too. For example,

• The original location of the file in SharePoint.
• The audit trail information about the object.
• The security access settings.
• Some of the metadata, (creation date, original author, version number, project name, signatory information, etc.)
• All versions and/or renditions of the document.

Having this information derived from SharePoint allows you to do the following:

• Identify that the content in the ECM system was originally imported from a specific SharePoint instance.
• Identify that a piece of content in the SharePoint document library has been archived at some point and where it was archived to.
• Retain the entire version or rendition history of an archived item.
• Maintain the security permissions model between systems.

In fact, you can pretty much bring anything across from SharePoint to the ECM system so long as you have a way of extracting it from SharePoint and then storing it in the ECM system. You might even collate information from other sources during the archive process and include that in the archived content.

Unfortunately, the most common approaches to this architecture will typically implement a "once only" and also "a one-way street" model. Content is moved over to the ECM system and the publish process is then finished. The publishing is a one-off process triggered by a change in state of the document or an external event. In some scenarios this is not a problem, for example when publishing to a records/retention archive or for eDiscovery collection when the content published to the archive will not be changed. In order to avoid this limitation you would have to have agents running on both the SharePoint and ECM systems that would monitor changes to the source or target objects and would be able to reconcile changes to either the source document or the archived copies...not a trivial task.

Conclusion

Again, this very loosely coupled model leaves a lot to be desired – the good news is that you end up with content in the most appropriate place at the most appropriate time but it is typically implemented as a one-off process. Even if you have a post-migration synchronization process in place you are really just moving content from SharePoint in to the ECM system...there has to be a better solution...doesn't there? Actually, this architecture comes in to its own when you combine it with some of the architectures that follow. After I've described all seven architectures I will focus on how to start combining them to get real efficiencies...the suspense is killing you - admit it.

One other consideration that you might notice is that you are potentially storing each piece of content multiple times which is a disk space and compliance liability.