MemorableURL.com

Spontaneous Thought Memorialized

We had a writer’s workshop a couple of weeks ago here at EMC. It was a great idea; we had a one day summit for internal and external authors, bloggers, wiki contributors, journalists and other general media contributors. We spent one evening and the whole next day not just discussing the intricacies, trials and tribulations of writing but also discussing some of the new concepts that are affecting our industry. It turned out to be a very interesting and informative day.

Barclay Blair, (the co-author of Information Nation), mentioned that blogs were dangerous because they represented “Spontaneous Thought Memorialized”. I looked at my Blogs and feel that mine are more like "a photograph of verbal diarrhea". This is probably why Barclay’s book is a best seller and mine…anyway.

I’ll be writing some more entries based on some of the sessions at the workshop over the next few weeks; look for articles on HSM across the cloud and maybe even some thoughts on Web 2.0.

Reference Architecture 6: Passive Back-end Aggregation

Read more about the eight reference architectures.

Let me preface this entry by saying that this reference architecture is similar to reference architecture #1 insomuch as it is not something that I recommend or endorse but it is something that I see in use fairly frequently today; in fact many of the solutions recommended by Microsoft fall in to this category. Best case, feel free to use this approach while you wait for more appropriate solutions to come along.

In the previous architectures we were trying to create a unified model so that an end user could simultaneously see content that resides in the SharePoint systems and also the traditional ECM systems. In this, (and the next), architecture we are going to directly deal with the SharePoint "silo sprawl" in the data center. I call this approach 'aggregation' not unification. The advantage of architectures 6 & 7 is that they are completely transparent to the end user - that's also their failing; these architectures are so transparent that you cannot use them to expose the advanced capabilities of the underlying ECM solutions to the end users. FYI - I am working on a cocktail of solutions that give you the best of both of these worlds but I'll probably keep that under wraps until I have it ironed out a lot more.

The general premise here is to leave each instance of SharePoint running independently as individual disparate systems but to create a virtual aggregation of the content and potentially the metadata behind the scenes. This approach does not actually change any part of SharePoint, the reference architecture's processes and technologies simply monitor the SharePoint implementations and create an aggregated view of the content stored across all libraries. This aggregated view can then be used to make security decisions, perform risk analysis, monitor file usage, etc.

For example, one could create a master index of all content in all SharePoint instances inside of the organization. This master index would contain the basic information about the contents of all of the document libraries along with the security access settings and a hash value of each of the pieces of content. You could then analyze this information and look for instances where a file’s security is not consistent. Let's consider a use case: assume that the company’s year-end financial results are stored in the finance department’s SharePoint document library with very restrictive security in the weeks before the official financial results are announced. If the passive unification process looked in the master repository index and noticed that this same file was also available on one of the finance department's collaborative SharePoint sites with no security then you might want to do something about that! Note that it would know that the two files were the same file even if the file name had been changed because the hash value would uniquely identify the file.

There are plenty of other use-cases for what to do with this data once you have access - monitoring data duplication, managing disposition, etc. but is this really an aggregated  back-end? It isn't really because nothing is actually aggregated in reality but it can be useful because it creates a virtual unified view of the data and once again it is a very non-invasive, low impact approach to the solution.

One issue to consider with this approach is scalability. If you have a billion data objects in your SharePoint systems then your enterprise-wide content index would have a billion entries in it. The other consideration is how to keep this index synchronized as content is added, altered and especially removed from the systems. Those types of operations are very CPU-heavy. 

Talk to the market leaders in database, search and indexing technology for advice on these issues. A billion objects synchronized across an enterprise is theoretically within the abilities of most enterprise database systems however I have talked to customers who generate over 1.5 billion records a day, (yes, that's the right number), these are transactional records currently ingested in to specialized storage but they could easily be stored as XML output files. Assume that they generate 1.5 billion records each working day they will end up with roughly 390 billion objects a year! Scalability becomes an issue very quickly with this kind of ingestion.

Again, I see this as being an architecture that is included for completeness and because it contains some concepts that are useful elsewhere, (think eDiscovery for example). Also, it can be "done on the cheap"; you can achieve passive virtual aggregation using the tools that you own today. A more useful reference architecture is to create some form of real unification at the back end...see the next reference architecture for more on that thrilling concept.

SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution.

Read more about the eight reference architectures. 

In the second of the eight architectures I take the general premise from the previous example and extend it. The content still lives natively in SharePoint while you are collaborating on it and is then "migrated" in to the traditional ECM system at a predetermined point in time.

As a matter of interest according to recent research, for most customers this movement of content from SharePoint to an ECM systems occurs because of one of the following reasons. The document:

• Becomes a formal business record.
• Is being collected as part of a legal discovery process.
• Needs to have extra protection applied, the application of digital rights management for example.
• Is being archived for operational reasons, e.g. the document library is no longer operational and can be archived in its entirety.
• Should be made available for re-use and re-purposing; perhaps being included in a formal structured business process.
• Will be published to web site or an external multi-channel device.

In this reference architecture the migration is done using an automated process which “publishes” the content from SharePoint in to the traditional ECM system. Fundamentally, the automated process is simply extracting the content from SharePoint, caching it locally and then uploading it in to your ECM system but the fact that the migration is controlled by a process not a person allows you to more easily perform some auxiliary tasks. For example, when the content is moved across from SharePoint it is possible to bundle up some of the object’s history, metadata and related content and to move that across too. For example,

• The original location of the file in SharePoint.
• The audit trail information about the object.
• The security access settings.
• Some of the metadata, (creation date, original author, version number, project name, signatory information, etc.)
• All versions and/or renditions of the document.

Having this information derived from SharePoint allows you to do the following:

• Identify that the content in the ECM system was originally imported from a specific SharePoint instance.
• Identify that a piece of content in the SharePoint document library has been archived at some point and where it was archived to.
• Retain the entire version or rendition history of an archived item.
• Maintain the security permissions model between systems.

In fact, you can pretty much bring anything across from SharePoint to the ECM system so long as you have a way of extracting it from SharePoint and then storing it in the ECM system. You might even collate information from other sources during the archive process and include that in the archived content.

Unfortunately, the most common approaches to this architecture will typically implement a "once only" and also "a one-way street" model. Content is moved over to the ECM system and the publish process is then finished. The publishing is a one-off process triggered by a change in state of the document or an external event. In some scenarios this is not a problem, for example when publishing to a records/retention archive or for eDiscovery collection when the content published to the archive will not be changed. In order to avoid this limitation you would have to have agents running on both the SharePoint and ECM systems that would monitor changes to the source or target objects and would be able to reconcile changes to either the source document or the archived copies...not a trivial task.

Conclusion

Again, this very loosely coupled model leaves a lot to be desired – the good news is that you end up with content in the most appropriate place at the most appropriate time but it is typically implemented as a one-off process. Even if you have a post-migration synchronization process in place you are really just moving content from SharePoint in to the ECM system...there has to be a better solution...doesn't there? Actually, this architecture comes in to its own when you combine it with some of the architectures that follow. After I've described all seven architectures I will focus on how to start combining them to get real efficiencies...the suspense is killing you - admit it.

One other consideration that you might notice is that you are potentially storing each piece of content multiple times which is a disk space and compliance liability.

SharePoint Archiving Assumptions...

One of the great things about blogging is that you have to think hard about what you are writing, literally tens of thousands of people might read your blog...granted I am lucky if half of my immediate family read this drivel but conceptually you get the point. Realizing that once you've written something you cannot retract it really does make you think twice about what you say.

Engaging my brain before I speak is not my natural state, just in case you wondered so this is quite a challenge for me.

Point in case - I'm sat on a flight back home typing the "SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution" entry and I realize that I've forgotten to mention one key point in my SharePoint-related ramblings...I am focused on the archiving and de-duplication primarily of unstructured content...PDF files, Word documents, etc.  I am neglecting the oh so important structured data types that SharePoint manages - discussion threads, calendar entries, etc. There is no doubt that these structured data types - especially the former example - are critical components of a compliant archiving solution.

Given this I'll do two things,

• Firstly I'll be explicit when I am referring specifically to archiving, unification and aggregation of unstructured content vs. unstructured content
• Secondly, I'll talk to the people from whom I steal all of my ideas and trick them in to telling me how to deal with archiving structured data. I'll then post some entries related specifically to that topic.

I'm pretty sure that the core compliance issues are fundamentally the same no matter the data type but as usual, the devil is in the details...

SharePoint-ECM Reference Architecture 1: Keep Systems Separate, Restrict Usage.

Read more about the eight reference architectures. 

Want to start this blog with an important caveat. The first reference architecture that we will look at is not something that I would advocate normally. It is something that I would refer to as Hobson's choice, (for those of you who don't know what that is click here), however I have included it for completeness as it is an architecture that we see being used readily inside companies today.  As I mentioned earlier, it is used primarily because there are no viable alternatives. So what is it...?

In this architecture, an end user moves a piece content from SharePoint into the traditional ECM system by manually exporting the object from SharePoint on to their desktop and then importing that object back in to the Enterprise Content Management system as a net new object. Equally, if a user needs to get content between two SharePoint sites they will follow the same procedure, i.e. they will export the content from one SharePoint site out onto the desktop and then import it into a new SharePoint site.

Depending on how this architectural used is used it can either be viewed as being extremely effective or very unruly, expensive and potentially dangerous. That said, it is cheap, easy to deploy and flexible enough for almost all of your user’s needs! Also, you're using SharePoint to manage content that is work in progress and your ECM system to store your copy of record - from that perspective the architecture looks good...from all the perspectives it sucks IMHO.

In this reference architecture you will have a multitude of SharePoint instances implemented in your organization as well as separate instances of your various traditional ECM systems. There is no technological link between the systems, each runs independently. Let me be very clear about one thing, this is not unification, it is not aggregation and it is not integration.

So, could you legitimately use this architecture and expect it to stand up to the rigors of modern-day usage? I suppose you could, you'd have to put some very strict processes in place and ensure that they were being followed. However even if you could ensure the content was being moved between systems at the right time by the right people you'd still has some significant issues. Consider the following:

The document in the ECM system is simply a copy of the final document that came out of the SharePoint system. You lose all of the rich history, metadata and audit information from the original document. You have no idea about the versions the preceded the final document, the comments made in it or the discussion threads created to support its creation. You also lose the connection back to the source document; if you make a copy of the “final” document and load it in to the ECM system you sever any ties back to the source document. If someone finds a mistake in the source document they have no way of knowing which ECM systems might contain copies of that “official” document.

Most importantly, you will not have controls in place to allow for the decommissioning of the plethora of SharePoint instances after the collaboration is compete. You have no way of knowing if critical content has actually been archived to the system of record and hence you cannot safely archive or destroy the library or site. Being able to archive or delete obsolete SharePoint instances is critical to prevent you from being left with thousands of SharePoint instances full of uncontrolled content lying around the organization acting as compliance and disk-usage nightmare.

So, architecture number one is not an architecture that we would like to see used widely by companies today however we should recognize that it is an architecture that we do see being used by companies all to frequently. The next six architectures that will look at represent much more robust, long-term and effective approach to aggregating or unifying SharePoint with a traditional enterprise content management system.

Integrating SharePoint with Traditional ECM Systems - Seven reference architectures...

Apologies for the brief Blog hiatus but I've been busy working on something that I think will be of great interest to many of you; how to allow you to yield the benefits of integrating SharePoint and your traditional ECM systems. In fact, I am now heading up a new group at EMC which is tasked with creating the SharePoint integration strategy and developing the necessary products to support it. I’ll also be working closely with partners who are developing SharePoint-based solutions for EMC – if you fit in to this category then feel free to contact me with details of your specific offerings.

Many customers ask how we can help make their SharePoint and Documentum systems cooperate. Not surprisingly, their users love the ease of use of SharePoint; the intuitive UI, the integrations in to the Office products and the collaborative tools. Their IT and Governance teams however may have real concerns about the pervasive nature of SharePoint.

In order to address the needs of our customers we need to be able to provide an environment where the SharePoint and their traditional ECM systems are unified, integrated or aggregated. I have come up with seven different concepts to address unifying or aggregating SharePoint with traditional ECM systems. Over the next couple of months I will address each of the architectures in detail including the pros and cons, example operations & cost/impact analysis details. At the end I’ll include a set of postings that talk about the real value – how to combine these reference architectures in to real-life solutions to your potential integration and content management challenges.

I am hoping that coinciding with the final postings I will have completed a new book that goes in to greater detail behind each of the architectures as well as many more use-cases, some issue analysis and general background information.

Let’s open with a brief overview of the seven reference architectures plus some of the general premises. Please note that over the next 2 months some of the subtleties behind each approach will change as the concepts mature.

Unification vs. Aggregation vs. Integration

I think that it helps to differentiate between how I use integration, unification and aggregation. Unification is the process of bringing together a set of disparate systems – getting SharePoint to work seamlessly with your traditional ECM systems is unification. Aggregation is the process of bringing together all of the content from one system in to logically sensible blocks, e.g. creating an environment where the content from your 12,000 SharePoint sites is stored in a single subsystem. I use integration as a term to general mean "making the systems work together".

Reference Architecture 1: Keep Systems Separate, Restrict Usage.

In this architecture the publishing of content between SharePoint and the Enterprise Content Management system is performed manually by the end user. This is not an approach that I’d advocate but it is included for completeness and because it is a method frequently used today simply out of necessity.

Reference Architecture 2: Loosely Coupled Solution

This is probably the key architecture behind any real unification solution today. In this architecture, content is moved between systems depending on its value to the organization. For example, work in progress might live in SharePoint and formal records might live in your ECM system. The move between systems is performed by a process and can be invoked manually or triggered by a specific event.

Reference Architecture 3: Use SharePoint as a Portal Container

In this commonly seen architecture, SharePoint hosts Web Parts that point at your SharePoint repository or at your Enterprise Content Management system. For example, one part of the screen’s real estate is displaying content from SharePoint while another part is displaying your ECM system’s Inbox. This is stretching the definition of “unification” IMHO but for reasons that will become clear later I think that this approach is often essential to make up a complete solution.

Reference Architecture 4: Passive Unification in Web Part

In reference architecture 3 the unification occurs within the SharePoint portal container. In this architecture the unification occurs within a specific Web Part; a single Web Part displaying content from a variety of disparate systems. We will look at this unification model from a passive perspective – only allowing the user to view or export the content.

Reference Architecture 5: Active Unification in Web Part

Building on reference architecture 4, we will see what happens if you attempt to add more complex operations to the affray; supporting operations like checkout, managing versions, connecting objects to a workflow, etc. The addition of these more complex operations increases the complexity exponentially.

Reference Architecture 6: Passive Back-end Aggregation

The first five architectures are all focused on unification. This penultimate example creates a passive aggregation model where we use tools to create a virtual aggregation across the SharePoint and traditional ECM environments. We can then monitor the aggregated environment and take actions as required. For example, we might compare the access rights on a piece of content that lives in 10 different document libraries and report on access inconsistencies.

Reference Architecture 7: Active Back-end Aggregation

The Holy Grail or Pandora’s Box? Real aggregation of the SharePoint and traditional ECM silos allows us to perform many operations that would be very difficult or even impossible otherwise. Having all objects from all environments addressable within a single space and supporting operations on those objects allows us to really address some of the current limitations in a scalable & secure way – think “single instance storage” for example.

Unification at the Middleware Layer

You may have noticed that the integrations above all manifest themselves in the client or server layer however the majority of SharePoint’s strength lies in the “middleware” layer. I am not ready to discuss middleware-level integrations yet but I expect to be able to add some more meat in to that layer of the sandwich by the time the book is ready to go to press…you may have to part with some cash to learn those bits!

Got comments or questions? Disgusted or in complete agreement? Let me know...

Sharepoint Records Management Update - Post ARMA

Firstly, I want to take my hat off to the team at the Microsoft booth at ARMA last week - I expected a less than welcoming response from them and I could not have been more wrong. I spent a considerable amount of time with members of their team at ARMA and was pleasantly surprised with the reaction, responses and rational that I got from them.

As you know, my concerns around Sharepoint are that its rapid adoption and pervasive nature create a compliance and general management liability for companies deploying it. Microsoft are quite rightly very pleased with the adoption rate for SharePoint 2007, my understanding is that Sharepoint is the most rapidly deployed Microsoft product ever.

As a direct result of the patience of one specific Microsoft employee, (who may want to remain anonymous), I think I get it the Sharepoint ethos now. (As a reward for his hard work I'll start capitalizing the P in SharePoint in postings from now onwards J.) I believe that the ethos is that SharePoint is orders of magnitude better than a network file share. I believe that this is the root cause of the gulf between my expectations and SharePoint's reality. Think about it - from a compliance perspective, if you compare SharePoint to your M: drive it is a huge leap forward; however, if you compare SharePoint to a mature enterprise content management system then it has significant catching up to do in some areas.

Microsoft have taken a very 'end-user' centric approach to the solution and the more I see of the UI the more impressed I am. It has little to do with compliance but I have to mention the "who might have the answer" approach to search that SharePoint exposes. If you do a search for a concept SharePoint not only gives you a results set back, it can also give you a list of people in your social network who are experts in that area. I know it is off topic but you have to admire the concept!

The guys that I spoke to did not have a defensive attitude when I put my concerns to them. Admittedly, they did not tell me that my concerns were unfounded but they made some great points.

• Uncontrolled growth: I put to them that SharePoint installations are growing uncontrolled inside of companies. They posed an interesting question - who's fault is that? They claim that Microsoft have a plethora of best-practice documents and always offer their services to aid companies with their deployment. Frankly, I think that Microsoft have an onus to also provide core functionality to aid in the management of deployments but I agree that their customers are big boys and that they should know better.
• Lack of unified back end: I discussed the lack of a unifying architecture with them too. In this area they had less of a story other than the fact that the level of unification can also be controlled by the customer based on their deployment method. WSS (Windows deployments will always lack any native back end unification but given that they directly replace a network file share, are you any worse off than you were with your M: drive? A MOSS installation will at least have the potential for cross site visibility and management although it seems to be sadly lacking at this point in time, (btw, if I am wrong and you know of a unifying technology from Microsoft or a partner then please do everyone a favor and post a response).

So what did we agree on? When deployed without good planning and controls SharePoint can create a potential liability for customers. I've identified the potential issues that this can cause but I thought it might help if I described some of the ramifications of one of these issues.

Conclusion

So I don't entirely agree with Microsoft's approach to 'enterprise' content management and I certainly don't like the fact that the records management capabilities look like an afterthought. However, they've made access to the core content management library services a very attractive proposition. People are a lot less likely to rebel against content management when it is so well integrated in to their desktop applications and so well exposed in a collaborative environment.

If we can find ways to unify the SharePoint usability with the scalability and compliance features of a true enterprise-strength content management system then I think we have real potential to win here. As the amount of content continues to grow our solutions need to be more pervasive and less invasive, I'd say that SharePoint goes a long way to aiding in that goal.

An open invitation to Microsoft...

The meeting that I had scheduled today with Microsoft was postponed, I'll work with them to set up another meeting but in the interim I'd like to re-state my opinion of Sharepoint and offer an open invitation to Microsoft...

Sharepoint bashing?

Let me be perfectly clear about my attitude towards Sharepoint. I have nothing against it as a product, in fact it is very compelling. My concerns are about how it is being deployed by companies today. Just because I work for a company that competes in some areas with Microsoft does not automatically mean that I am taking a competitive position. Let me explain:

• Why not just let people use Sharepoint and then worry about compliance later? I've just spent the better part of 12 months writing a book that focuses on how to take a holistic view of implementing compliance solutions, how it pays to think about compliance up front and throughout all of your systems. This is a passion of mine and I cannot in good faith watch quietly while companies implement a technology without thinking about compliance. If you don't care then have at it and implement Sharepoint throughout the organization, let's face it, it is a hugely compelling solution.
• Why are you picking on Sharepoint and not talking about other collaborative environments? Simply because it is so successful. If no one used it then I would not care, if it was a point solution for a single industry then I would not care as much but it isn't...that said, my comments stand for ANY system in your company not just Sharepoint. My advice is as always "Consider compliance considerations now or pay the price later", it is your choice but do not doubt that I will say, "I told you so" if you ignore me.
• If Sharepoint is just a collaboration tool then why worry about compliance? Ah Padawans...what are people collaborating on? If it is the office fantasy football league then you are fine but what if you are using Sharepoint to collaborate around business documents. These document could be formal records and they are almost certainly discoverable[1]. If they are records or get included in a discovery response then you'll have to be able to protect them. What if the documents are confidential, you might need to encrypt them. What if they contain private health information, you might need to protect or redact them...you get the idea.
• Would you rather people didn't use Sharepoint? Not really. For example, from a compliance perspective Sharepoint, if used correctly, is a much better location for content than a network file share - it allows you to create context around your content, you can store attributes, create relationships and use a more formal structured approach to storing your content. This context allows you to identify content that needs to be protected for compliance reasons.  Would I rather you used Documentum? Of course I would, I have stock in EMC but that's not my point here; my message isn't "don't use Sharepoint" my message is "if you are using Sharepoint then take these issues in to consideration". Think holistically today or pay the fines and remedial work costs later.
• You're on a bit of a downer, would anything make you happy? I'm naturally a bit manic but I'm also passionate, this combination might look negative but truthfully I just give a dam. I'm actually not down on Microsoft per se but I do think that when you release a product like Sharepoint hoping that it will pervade the entire commercial market globally then you have a onus - world domination has its responsibilities too you know. What would make me happy? I'd love Microsoft to start to take compliance more seriously, bake it in to the core product rather than relying on partners to retrofit it as an afterthought. I'd love to have an open debate about the best practices around managing compliant data in Sharepoint. I'd love to have this conversation with Microsoft partners as well as Microsoft - if you are looking for a business opportunity then set up a practice applying compliance controls to Sharepoint content, (to make lots of money focus on remedial work and just sit back and wait). I'd love Microsoft to ask my advice, I would not be shy in putting my opinions forward in person...in fact I'd like to formally make that offer:

An open invitation to Microsoft.

I think that it would benefit everyone, including Microsoft, if this discussion was two-sided. I think that everyone would love to hear what Microsoft have planned to help enterprises to make best use of Sharepoint. You name the format and the location and I'll do my best to accommodate. In fact, I'll be at ARMA in Baltimore next week - I'd love to meet and talk about this issue, I'll even give you a free copy of my book to read!

Seriously, I'm not trying to bate Microsoft, I am trying to represent the best interests of their customers and their partners.

An open invitation to the rest of you.

I'll be at ARMA next week, if you'd like to chat then either find me at the booth, at the OmniRIM launch party in the aquarium, come along to my session or email me ahead of time to arrange a meeting. My work email address is chapman_andrew(at)emc.com if you want to pre-arrange something. My session is Monday, October 8th,  2:00 p.m. - 3:15 p.m., it is entitled "The Truth About Federated Records Management – Be Careful What You Ask For!".

---

[1] A discovery is the action you take if when you get sued, the courts issue a subpoena telling you to produce all data/documents relating to the case and you have to locate them and lock them down for delivery to the court.

Microsoft Sharepoint and Records Management Opinions

I am meeting with Microsoft this week to discuss some of my concerns about enforcing compliance with content in Sharepoint repositories and to also talk through some of the potential solutions. It seems to me that there are a few different approaches to managing records that reside in Sharepoint Document Libraries. 

Firstly, let's recap the issue caused by the success of Sharepoint here. Some companies are letting Sharepoint instances be created with little control whatsoever. I got a lot of comments from my last Microsoft entry saying that companies don't have to do this but my point still stands - they are. I'll refer you once again to a customer I spoke to that expects to create 12,000 new document libraries, (aka Silos), a year and they have no way of being able to search, discover, hold, de-duplicate etc. across all of these repositories.

Secondly, let's recap the consequences of this. If you are dumping content in to these silos without any regard for how to apply compliance controls to the content then you are stepping back 10 years in compliance technology. Each one of these silos could contain a copy of a record - if it is the only copy then you have a legal obligation to retain it - if it is not the only copy then you have a liability because when you dispose of the record you may have left a copy of it behind and that's then discoverable to produce in court.

It seems to me that in general there are a number of different ways of addressing records management across these silos. I'm writing this before I meet with Microsoft for two reasons - one to get your feedback, (be quick though, I'm meeting them on Tuesday) and secondly to dump down my thoughts in preparation for the meeting. The methods that I can think of can be expressed in three general groups, manage in place, move out of Sharepoint or both, (keep a copy in Sharepoint and move a copy out too.)

Here's a very brief synopsis of each option and all of my questions, I'll expand on them over time and I'll give you the feedback from Microsoft later this week. Do you have any other options that spring to mind?

Option 1a - Manage in place in Sharepoint Document Library using Native Functionality

You are relying on Sharepoint's native capabilities to protect the content. Where will cross-library/cross-site functionality come from? What about de-duplication? How will you control true disposition across all libraries? What about storage management?

Option 1b - Manage in place in Sharepoint Document Library using Federated Model 

In this model you will still rely on Sharepoint's native capabilities to secure the content which is a risk. However, you'll also have a central repository that is tracking all records from a single place. This might supplement 1a with some enterprise wide functionality like search, track, de-duplicate. It would facilitate true disposition of all copies and would be a boon for discovery.

Option 2a - Move from Document Library to Secure Records Repository

This gives the best records management capabilities - you are taking the content out of the Sharepoint library and putting it in to a system specifically designed for records management. There's no doubt in my mind that this gives you the ultimate records management control but it is very invasive. If someone happened to be using the file they will be less that delighted when it simply disappears just because it was deemed to be a record one day.

Option 2b - Move from Document Library to Secure Records Repository, (leave link behind)

On paper, theoretically, in principal this rocks. Unfortunately, in reality it is less that optimal. I'm going to say more about the pros and cons of this one later but for now just consider that you are at the mercy of the capabilities of the link. Let me give you just one nasty example of what you might want to do:

The link that you leave behind needs to act like the original document for the benefit of anyone working in the document library. However, the link represents a record now so what do you do if someone goes to edit the link?

1. Don't let them edit the file.
2. Let them open the file read-only and then save it manually as a new object.
3. Let them open the file for editing and save it over the old version but in the background maintain the 'record copy' in the secure repository as the copy of record, (maybe at least informing the user that you did it?)

Which one works best? I like the last one because it is non-invasive and cool but it is probably confusing to the end user and the records team might not like it. Oh, and the link probably needs to work bi-directionally so if someone deletes the record from the secure repository should you delete the link even it is in use or should you put a copy of the file back on the local system for convenience, (which is far from records management disposition by anyone's standard).

Option 3 - Copy from Document Library to Secure Records Repository

Oh heavens to Betsy. Copy a record...please. I was with a customer last week who has 600 terabytes of data on their network fileshares - assume that 5% of those files are records...that's...well it is a lot of duplicated data that's what and don't even talk to me about disposition... 

...anything else?

Got any questions or other options for me to think about? I'll start to focus on some of the details behind some of the more realistic options soon. Watch this space.

An open invitation to the rest of you minions...ARMA 2007

I'll be at ARMA next week in Baltimore. If you'd like to chat then either find me at the EMC booth, buy me a drink at the OmniRIM launch party in the aquarium, come along to my session or email me ahead of time to arrange a meeting.

My work email address is chapman_andrew(at)emc.com if you want to pre-arrange something.

My session is Monday, October 8th,  2:00 p.m. - 3:15 p.m., it is entitled "The Truth About Federated Records Management – Be Careful What You Ask For!". I don't know where the session is but I do know that there will be free ice cream.