MemorableURL.com

Managing Social Data: Is SharePoint the Answer?

I was approached late last year by ARMA to write an article for their Information Management publication. During a lapse of good judgment I suggested that a topic that everyone is interested in but no one is talking about is how to apply compliance controls to ‘semi-structured data’.  Semi-structured data being those things like Blogs, Wikis, activity feeds, calendar items, etc. that are stored as structured data but look very similar to unstructured data when being consumed. 

Here's the intro to the article and the cover of the magazine. Before you ask, no that's not me hiding behind the question mark.

Managing Social Data:
Is SharePoint the Answer?

Records management (RM) professionals have been challenged to manage electronic data for some time. Their efforts have tended to focus on unstructured data, such as documents, scanned images, and spreadsheets. Recently, structured data held in databases has also come under close scrutiny, especially during discovery. However, there is a third category of data that tends to be overlooked. Read on for more or download the PDF version

Although users of all semi-structured data providers need to take heed, I pointed out that SharePoint was an opportunity for companies to look at a single point to apply these controls. If you are using SharePoint for all of your ‘Web 2.0’-type content then why not look to SharePoint to apply the compliance controls that you need.

Interestingly, when I was 80% of the way through writing the document I reached the conclusion that all I was going to provide you my dear readers was the bad news; I was not unearthing much in the way of solutions to these issues. Luckily, I was able to get some time from some of the SharePoint product team and they were able to add some of the color around the capabilities of SharePoint 2010 - thanks to all concerned. I’d also like to thank the team from Gimmal who provided some of the use cases and real-life background data.

The article is long…maybe too long but it should be worth a read if you care about SharePoint, compliance or have chronic insomnia. This is a topic which will only get worse before we catch up...

Main article: http://content.arma.org/IMM/March-April2010/IMM0310managingsocialdata.aspx

PDF Version: http://content.arma.org/IMM/Libraries/March-April_2010_PDFs/IMM_0310_managing_social_data.sflb.ashx

Note: My email address is quoted incorrectly in the magazine, it is actually [email protected] (with an underscore not a period). My fault not ARMA's.

 

Managing Semi-structured data in SharePoint and what’s new for ECM fans in SharePoint 2010

So, it may seem like I have forgotten how to write but in reality I’ve been writing, just not here! I have two papers complete which I think my usual readers will be interested in. One is for Information Management Magazine (ARMA’s publication) and the other for SharePointPro Connections Magazine. I need to wait until they are published before I can post links to, quote them or discuss them in detail but here’s a brief synopsis of each of them just to get you all excited:

Information Management (ARMA)

As well as publishing IM magazine ARMA represents information management professionals and records managers. They asked me to write an article about how records managers could manage the new wave of social data types – Blogs, Wikis, IM, Activity feeds, polls, discussion threads, etc. Given that SharePoint purports to manage most of these they also asked that the article would have a SharePoint focus.

This one started out well, I defined the data types and the issues at hand. I initially called this type of data ‘semi-structured’ but it turned out that the DB world already had dibs on that phrase so after talking to Microsoft I coined/stole the phrase ‘social data’. Technically what I analyzed was data that was structured at the back-end but had an associated presentation layer that makes it look unstructured to an end user.

It all fell apart when I failed to find anyone with an ‘off the shelf’ solution to the problem. Thankfully I had some great conversations with the SharePoint product team and it turns out that SharePoint 2010 does include some basic capabilities. I stitched that together with some of the custom solutions that I discovered and it all came together rather well.

SharePointPro Connections Magazine

This article is a summary of what’s new in SharePoint 2010 from an ECM perspective – the article highlights the key content management-related features and questions how well they might address enterprise requirements.  I cover most of the new ECM-related features and dig down into some of them where I feel there’s either a misunderstanding wrt what the feature really does or where I think Microsoft missed the mark. IMHO this is not as interesting as the ARMA article which digs into new territory but it is a good primer for SharePoint 2010.

These types of articles frequently get me into trouble because SharePoint fans think that I am bashing SharePoint and my Documentum colleagues think that I am glorifying SharePoint. My solution is to take a truly agnostic view of the factors and try to be honest. I am not known for my tact but this does seem to be working so I’ll stick with it. I’ve also taken to talking to insiders at Microsoft to make sure that what I am saying is at least fair – maybe not palatable but fair.

I’ll post links to the articles as they get published…

How do I know whether I *need* a traditional ECM system with SharePoint?

I was speaking at an AIIM conference in Chicago yesterday (http://www.aiim.org/ecmseminar/). The topic of my talk was how traditional ECM systems add value to SharePoint. I rambled on about all of the value adds and at the end, someone asked me how they would know whether SharePoint was good enough as-is or whether they needed to supplement it with an ECM system. I thought that this was a great question…

The answer might be, “if you do not know whether you need one then you probably don’t” but the answer is more accurately “if you do not know whether you need one then you probably don’t…yet”. If you are a small company rolling out SharePoint, or a large company doing a limited pilot then you are probably doing a great job managing the process. There may come a time however when you will start to think about long-term archiving of that content or retention management. IT might want to start to implement HSM or to de-duplicate the content. You might want to start to intelligently retain version information, etc…

My answer at the conference was to say that if everything was ticking along nicely today then you are probably OK but what you absolutely should do is think about the future aggregation and intelligent management of those sites today. Here’s the deal…would you allow each department in your organization to roll out their own Microsoft Exchange mail server with its own domain and their own email addresses? Of course not because any future reconciliation of those systems would be a nightmare. Then treat your SharePoint implementations in the same way. I told audience to assume that they would bring the systems together in the future and to that end they should do a few simple things:

· Make sure that there are no maverick deployments of SharePoint in the organization. All SharePoint sites have to be managed by IT – no exceptions.

· Make sure that the user identity management is common across all sites, (probably the most important one.)

· Whenever possible manage the creation of object types especially ensuring that common attributes have the same name and draw their values from a common domain list, (don’t have a project number represented in one site as Proj_# and in another as ProjectNo and certainly don’t make them free text fields!)

· Consider how you would aggregate the sites moving forwards…is there anything else that you might be able to do to make this easier moving forwards?

Here’s a cast iron Chapman guarantee. If you follow the steps above you will reap the rewards even if you never aggregate the content or work with a traditional ECM system. Don’t believe me? Just consider how much easier eDiscovery would be if you followed the steps above? Need to find all of the drivel that Andrew Chapman wrote? If I have the same user identity in all SharePoint instances then that just became a lot easier.

Spontaneous Thought Memorialized

We had a writer’s workshop a couple of weeks ago here at EMC. It was a great idea; we had a one day summit for internal and external authors, bloggers, wiki contributors, journalists and other general media contributors. We spent one evening and the whole next day not just discussing the intricacies, trials and tribulations of writing but also discussing some of the new concepts that are affecting our industry. It turned out to be a very interesting and informative day.

Barclay Blair, (the co-author of Information Nation), mentioned that blogs were dangerous because they represented “Spontaneous Thought Memorialized”. I looked at my Blogs and feel that mine are more like "a photograph of verbal diarrhea". This is probably why Barclay’s book is a best seller and mine…anyway.

I’ll be writing some more entries based on some of the sessions at the workshop over the next few weeks; look for articles on HSM across the cloud and maybe even some thoughts on Web 2.0.

SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution.

Read more about the eight reference architectures. 

In the second of the eight architectures I take the general premise from the previous example and extend it. The content still lives natively in SharePoint while you are collaborating on it and is then "migrated" in to the traditional ECM system at a predetermined point in time.

As a matter of interest according to recent research, for most customers this movement of content from SharePoint to an ECM systems occurs because of one of the following reasons. The document:

• Becomes a formal business record.
• Is being collected as part of a legal discovery process.
• Needs to have extra protection applied, the application of digital rights management for example.
• Is being archived for operational reasons, e.g. the document library is no longer operational and can be archived in its entirety.
• Should be made available for re-use and re-purposing; perhaps being included in a formal structured business process.
• Will be published to web site or an external multi-channel device.

In this reference architecture the migration is done using an automated process which “publishes” the content from SharePoint in to the traditional ECM system. Fundamentally, the automated process is simply extracting the content from SharePoint, caching it locally and then uploading it in to your ECM system but the fact that the migration is controlled by a process not a person allows you to more easily perform some auxiliary tasks. For example, when the content is moved across from SharePoint it is possible to bundle up some of the object’s history, metadata and related content and to move that across too. For example,

• The original location of the file in SharePoint.
• The audit trail information about the object.
• The security access settings.
• Some of the metadata, (creation date, original author, version number, project name, signatory information, etc.)
• All versions and/or renditions of the document.

Having this information derived from SharePoint allows you to do the following:

• Identify that the content in the ECM system was originally imported from a specific SharePoint instance.
• Identify that a piece of content in the SharePoint document library has been archived at some point and where it was archived to.
• Retain the entire version or rendition history of an archived item.
• Maintain the security permissions model between systems.

In fact, you can pretty much bring anything across from SharePoint to the ECM system so long as you have a way of extracting it from SharePoint and then storing it in the ECM system. You might even collate information from other sources during the archive process and include that in the archived content.

Unfortunately, the most common approaches to this architecture will typically implement a "once only" and also "a one-way street" model. Content is moved over to the ECM system and the publish process is then finished. The publishing is a one-off process triggered by a change in state of the document or an external event. In some scenarios this is not a problem, for example when publishing to a records/retention archive or for eDiscovery collection when the content published to the archive will not be changed. In order to avoid this limitation you would have to have agents running on both the SharePoint and ECM systems that would monitor changes to the source or target objects and would be able to reconcile changes to either the source document or the archived copies...not a trivial task.

Conclusion

Again, this very loosely coupled model leaves a lot to be desired – the good news is that you end up with content in the most appropriate place at the most appropriate time but it is typically implemented as a one-off process. Even if you have a post-migration synchronization process in place you are really just moving content from SharePoint in to the ECM system...there has to be a better solution...doesn't there? Actually, this architecture comes in to its own when you combine it with some of the architectures that follow. After I've described all seven architectures I will focus on how to start combining them to get real efficiencies...the suspense is killing you - admit it.

One other consideration that you might notice is that you are potentially storing each piece of content multiple times which is a disk space and compliance liability.

SharePoint Archiving Assumptions...

One of the great things about blogging is that you have to think hard about what you are writing, literally tens of thousands of people might read your blog...granted I am lucky if half of my immediate family read this drivel but conceptually you get the point. Realizing that once you've written something you cannot retract it really does make you think twice about what you say.

Engaging my brain before I speak is not my natural state, just in case you wondered so this is quite a challenge for me.

Point in case - I'm sat on a flight back home typing the "SharePoint-ECM Reference Architecture 2: Loosely Coupled Solution" entry and I realize that I've forgotten to mention one key point in my SharePoint-related ramblings...I am focused on the archiving and de-duplication primarily of unstructured content...PDF files, Word documents, etc.  I am neglecting the oh so important structured data types that SharePoint manages - discussion threads, calendar entries, etc. There is no doubt that these structured data types - especially the former example - are critical components of a compliant archiving solution.

Given this I'll do two things,

• Firstly I'll be explicit when I am referring specifically to archiving, unification and aggregation of unstructured content vs. unstructured content
• Secondly, I'll talk to the people from whom I steal all of my ideas and trick them in to telling me how to deal with archiving structured data. I'll then post some entries related specifically to that topic.

I'm pretty sure that the core compliance issues are fundamentally the same no matter the data type but as usual, the devil is in the details...

SharePoint-ECM Reference Architecture 1: Keep Systems Separate, Restrict Usage.

Read more about the eight reference architectures. 

Want to start this blog with an important caveat. The first reference architecture that we will look at is not something that I would advocate normally. It is something that I would refer to as Hobson's choice, (for those of you who don't know what that is click here), however I have included it for completeness as it is an architecture that we see being used readily inside companies today.  As I mentioned earlier, it is used primarily because there are no viable alternatives. So what is it...?

In this architecture, an end user moves a piece content from SharePoint into the traditional ECM system by manually exporting the object from SharePoint on to their desktop and then importing that object back in to the Enterprise Content Management system as a net new object. Equally, if a user needs to get content between two SharePoint sites they will follow the same procedure, i.e. they will export the content from one SharePoint site out onto the desktop and then import it into a new SharePoint site.

Depending on how this architectural used is used it can either be viewed as being extremely effective or very unruly, expensive and potentially dangerous. That said, it is cheap, easy to deploy and flexible enough for almost all of your user’s needs! Also, you're using SharePoint to manage content that is work in progress and your ECM system to store your copy of record - from that perspective the architecture looks good...from all the perspectives it sucks IMHO.

In this reference architecture you will have a multitude of SharePoint instances implemented in your organization as well as separate instances of your various traditional ECM systems. There is no technological link between the systems, each runs independently. Let me be very clear about one thing, this is not unification, it is not aggregation and it is not integration.

So, could you legitimately use this architecture and expect it to stand up to the rigors of modern-day usage? I suppose you could, you'd have to put some very strict processes in place and ensure that they were being followed. However even if you could ensure the content was being moved between systems at the right time by the right people you'd still has some significant issues. Consider the following:

The document in the ECM system is simply a copy of the final document that came out of the SharePoint system. You lose all of the rich history, metadata and audit information from the original document. You have no idea about the versions the preceded the final document, the comments made in it or the discussion threads created to support its creation. You also lose the connection back to the source document; if you make a copy of the “final” document and load it in to the ECM system you sever any ties back to the source document. If someone finds a mistake in the source document they have no way of knowing which ECM systems might contain copies of that “official” document.

Most importantly, you will not have controls in place to allow for the decommissioning of the plethora of SharePoint instances after the collaboration is compete. You have no way of knowing if critical content has actually been archived to the system of record and hence you cannot safely archive or destroy the library or site. Being able to archive or delete obsolete SharePoint instances is critical to prevent you from being left with thousands of SharePoint instances full of uncontrolled content lying around the organization acting as compliance and disk-usage nightmare.

So, architecture number one is not an architecture that we would like to see used widely by companies today however we should recognize that it is an architecture that we do see being used by companies all to frequently. The next six architectures that will look at represent much more robust, long-term and effective approach to aggregating or unifying SharePoint with a traditional enterprise content management system.

Integrating SharePoint with Traditional ECM Systems - Seven reference architectures...

Apologies for the brief Blog hiatus but I've been busy working on something that I think will be of great interest to many of you; how to allow you to yield the benefits of integrating SharePoint and your traditional ECM systems. In fact, I am now heading up a new group at EMC which is tasked with creating the SharePoint integration strategy and developing the necessary products to support it. I’ll also be working closely with partners who are developing SharePoint-based solutions for EMC – if you fit in to this category then feel free to contact me with details of your specific offerings.

Many customers ask how we can help make their SharePoint and Documentum systems cooperate. Not surprisingly, their users love the ease of use of SharePoint; the intuitive UI, the integrations in to the Office products and the collaborative tools. Their IT and Governance teams however may have real concerns about the pervasive nature of SharePoint.

In order to address the needs of our customers we need to be able to provide an environment where the SharePoint and their traditional ECM systems are unified, integrated or aggregated. I have come up with seven different concepts to address unifying or aggregating SharePoint with traditional ECM systems. Over the next couple of months I will address each of the architectures in detail including the pros and cons, example operations & cost/impact analysis details. At the end I’ll include a set of postings that talk about the real value – how to combine these reference architectures in to real-life solutions to your potential integration and content management challenges.

I am hoping that coinciding with the final postings I will have completed a new book that goes in to greater detail behind each of the architectures as well as many more use-cases, some issue analysis and general background information.

Let’s open with a brief overview of the seven reference architectures plus some of the general premises. Please note that over the next 2 months some of the subtleties behind each approach will change as the concepts mature.

Unification vs. Aggregation vs. Integration

I think that it helps to differentiate between how I use integration, unification and aggregation. Unification is the process of bringing together a set of disparate systems – getting SharePoint to work seamlessly with your traditional ECM systems is unification. Aggregation is the process of bringing together all of the content from one system in to logically sensible blocks, e.g. creating an environment where the content from your 12,000 SharePoint sites is stored in a single subsystem. I use integration as a term to general mean "making the systems work together".

Reference Architecture 1: Keep Systems Separate, Restrict Usage.

In this architecture the publishing of content between SharePoint and the Enterprise Content Management system is performed manually by the end user. This is not an approach that I’d advocate but it is included for completeness and because it is a method frequently used today simply out of necessity.

Reference Architecture 2: Loosely Coupled Solution

This is probably the key architecture behind any real unification solution today. In this architecture, content is moved between systems depending on its value to the organization. For example, work in progress might live in SharePoint and formal records might live in your ECM system. The move between systems is performed by a process and can be invoked manually or triggered by a specific event.

Reference Architecture 3: Use SharePoint as a Portal Container

In this commonly seen architecture, SharePoint hosts Web Parts that point at your SharePoint repository or at your Enterprise Content Management system. For example, one part of the screen’s real estate is displaying content from SharePoint while another part is displaying your ECM system’s Inbox. This is stretching the definition of “unification” IMHO but for reasons that will become clear later I think that this approach is often essential to make up a complete solution.

Reference Architecture 4: Passive Unification in Web Part

In reference architecture 3 the unification occurs within the SharePoint portal container. In this architecture the unification occurs within a specific Web Part; a single Web Part displaying content from a variety of disparate systems. We will look at this unification model from a passive perspective – only allowing the user to view or export the content.

Reference Architecture 5: Active Unification in Web Part

Building on reference architecture 4, we will see what happens if you attempt to add more complex operations to the affray; supporting operations like checkout, managing versions, connecting objects to a workflow, etc. The addition of these more complex operations increases the complexity exponentially.

Reference Architecture 6: Passive Back-end Aggregation

The first five architectures are all focused on unification. This penultimate example creates a passive aggregation model where we use tools to create a virtual aggregation across the SharePoint and traditional ECM environments. We can then monitor the aggregated environment and take actions as required. For example, we might compare the access rights on a piece of content that lives in 10 different document libraries and report on access inconsistencies.

Reference Architecture 7: Active Back-end Aggregation

The Holy Grail or Pandora’s Box? Real aggregation of the SharePoint and traditional ECM silos allows us to perform many operations that would be very difficult or even impossible otherwise. Having all objects from all environments addressable within a single space and supporting operations on those objects allows us to really address some of the current limitations in a scalable & secure way – think “single instance storage” for example.

Unification at the Middleware Layer

You may have noticed that the integrations above all manifest themselves in the client or server layer however the majority of SharePoint’s strength lies in the “middleware” layer. I am not ready to discuss middleware-level integrations yet but I expect to be able to add some more meat in to that layer of the sandwich by the time the book is ready to go to press…you may have to part with some cash to learn those bits!

Got comments or questions? Disgusted or in complete agreement? Let me know...

Structured/Database Data as a Record

I know that this blog entry is a bit long but you try explaining this kind of abstract concept in less words, it is hard!

I work for EMC, as well as making disk drives, (I'm a software guy so that's just exhausted my total knowledge of hardware), we are actually the 6th largest software company in the world. One the the most challenging things about this is understanding the capabilities of all of the other applications in the company -- more importantly one needs to be able to assess how the other applications can be used to increase the value of your specific software offering. Bear in mind that here at EMC we have exactly 7498 different software products, (OK I completely made that up but we do have a lot - it could easily be more than this!).

Occasionally it all comes together -- you have a epiphany and realize that you can take a couple of existing products and solve an age-old problem. It is amazingly satisfying when this happens. One relatively recent example of this is the "Structured Data as a Record" conundrum. Let me explain...

I used to get a lot of questions about storing 'structured data' as a record. To those of us who live in the enterprise content management space the term 'structured data' refers to data that comes from a relational database. Customers want to take data currently stored in a database and make parts of it in to a record. They want to do this without removing the content from the database. I used to think that this showed a misunderstanding of what a record is but now I realize that it actually highlights a classic problem with understanding software solutions. The customer was asking for a specific solution rather than defining the problem -- they were asking how to make database content immutable in place but that's a solution not a problem; the problem they are really trying to solve is how to produce the output of this data at a future time.

Actions on Structured Data

Typically people want to do 3 things to data in a database, (other than just use it of course). They want to optimize the database, archive the data and apply compliance controls to the data, (these are not mutually exclusive BTW, they may want to do all 3 at once). We have in-house software and partners galore that deal with the first two but the last one is a different animal. If you asked most people in the know how to do this they would tell you that it was not possible - with good cause. Before we dive in to the solution let's consider the three main types of structured data to which you typically need to apply compliance controls.

Types of Structured Data

Structured data is usually generated by computer applications and systems -- not people, it can be sub-divided in to page-oriented, line-oriented and native database.

Page-orientated Data is designed to to be printed in a page format, it is made to be easily read and interpreted by a human, the page breaks add context to the final report. It has what we call "print fidelity", this means that the layout is significant. Typical examples would be bank statements or house-sale closing documents.

Line-orientated Data is just as it sounds...line after line of boring data. It is typically in date/time sequence and may have a fixed or variable line structure. It may be character delimited, fixed column, undelimited lines of text or even theoretically XML, (XML is just delimited text anyway). This is not usually designed to be ready by people on a day to day basis. A good example are transaction logs from a trading system or audit logs from one of your enterprise systems.

Database Data is the stuff at rest in its native environment, it is sitting in your relational database of choice. To be clear, the two types above will often have been derived from a database in the first place.

So what's the problem?

Imagine trying to make a single record in a relational database immutable. I cannot even define what that means. Are you taking a row from a table and locking it down, or just a single value? Do you lock down the relationships between the data too? What about the queries and reports used to consolidate the values in to useable data?  The questions keep on going...

So what's the solution?

Back to the advantage of have access to all of the resources of a 13 billion dollar company then... EMC purchased a company called Acartus a few years ago. I always thought that their software was cool, it allowed you to grab a print run and instead of letting it stream to a printer the software converted it to a PDF file. You could then archive this PDF as an exact facsimile of the printed documents. In the case of page-oriented print run, the software knew where the physical and logical breaks in the run were so it knew where each page started and ended but also where each logical section was. It could take you to page 4 of a 15 page bank statement in a PDF that contained 132,000 statements.  When we bought Acartus they embarked on a project to use the EMC Documentum repository to natively store these documents. The final product is now called Archive Services for Reports -- ASR. 

Recently I started working with the ASR team on a specific account. We already knew that we could manage report output as a record but we realized that we could also use the capabilities of ASR and RM to address some of the structured content compliance challenges. I'll keep it brief, (well Chapman brief), but email me with your questions if you want to understand more and I'll add more detail to the blog.

Redefining "Structured Content"

Firstly, while the content is in the active database you really cannot make it in to a record. However, if you take the data out of the database you have a fighting chance, this sounds like a cop out but it isn't.

Consider the real requirement. The regulations do not say that your active database data need to be retained in situ, the regulations say that you must retain and be able to produce documentary evidence of a specific event. For example, you may need to guarantee to the courts or customers that you can provide details all of your customer's credit card transactions for 7 years.

So, the solution just needs to allow you to recover specific details drawn from the structured data. In order to do this using today's technology we need to scrub the attempt to manage data in place in the database and focus on meeting the regulatory requirements by using page- and line-oriented data.

Page-oriented Structured Data as a record

Let's return to the way too ubiquitous credit card statement. Your personal account details and the details of each transaction are stored in a relational database. Assume that the regulations state that you must keep those statements for 7 years.

If you tried to lock down the data in the native database you'd be heading for trouble. However, if you use something like ASR then you'd be able to store the entire day's credit card statements in a single uber-PDF file and retain it for 7 years. Sounds fairly trivial doesn't it? In fact, we can do this today without using a variety of products. Now consider the nightmare caused by "exceptions". What happens if a customer sues the credit card company over a specific transaction that happened 6 years ago?

The credit card company needs to place a legal hold on one of the 132,000 statements in that uber-PDF file. With most technologies you would have to hold all 132,000 statements until the legal hold was removed. This causes a huge liability to a company who would rather dispose of their data as soon as legally allowable.

The epiphany that we had was to realize that because ASR is more intelligent than the other report archiving solutions we could actually demonstrate a much more sophisticated approach. Out of the box, ASR allows you to check out a single logical section of a print run and view it as a single object separate from the uber-PDF. I request someone's statement and ASR extracts just those pages and creates a new object in the repository that only contains that individual statement. It then updates its internal index so that it knows that the logical section is now a separate object. The original data is left in the uber-PDF.

Assume that you need to place a legal hold on a single statement. Using ASR you find the statement and apply the hold, behind the scenes that single statement turns in to a new object separate from the original PDF file and it is that new object that has the hold applied to it. When the original 7 years are up you can safely delete the original uber-PDF file knowing that the held statement is protected. There's a little more to it than this to implement the house keeping but bottom line is that it allows you to do records management on the full print run by exception only.

Line-oriented Structured Data as a record

Line-oriented is a little different because the data does not have such a complex logical structure. In fact, each line typically represents a single discrete piece of information. The results of a transaction, a security log event, etc.

The key requirement with line-oriented data is typically to archive it because the production systems get very full very quickly. However, the data that is going to be archived is usually discoverable or more often required to defend an action.

Again, during our epiphany session we realized that ASR could be used to not only archive line-oriented data but also to enforce compliance controls to that same data. We archive content from the live database and print it to a PDF file. The PDF file might contain lines representing transactions from a single day -- this could be hundreds of thousands of lines of data. The PDF file is stored in the EMC Documentum repository until a discovery or an analysis needs to take place. At this point the contents of the PDF file are loaded in to a temporary database, you query the temporary database, get your data as a standalone XML file and then make that in to a record. The XML extract file is generated automatically by the system which means that it can be shown to be a good record of the original data -- we can effectively show a coherent chain of custody.

Conclusion

Does this actually solve the core problem of managing structured content as a record or is it just a software vendor using their tools to manufacture a half-baked pretend solution? I believe it truly is the former. It only looks like the latter if you forget the real objective -- to retain discoverable content in a way that shows a chain of custody.

It doesn't work in every case but believe me, it works a lot better in reality than it does in theory. In theory there are loads of reasons why you would not want to do this but in reality it is quite achievable.

As always, email me if you want more information... 

Federating Policies vs. Federating Records

There seems to be some confusion in the world of federated records management about what exactly is being federated. There are 2 distinct use cases what you might want to consider:

1) Federating just the records/retention policies. The use case here would be where you would manage the master policies in one system - the master policy management system. Each remote system looks at the master policy management system to get the policy information or the master policy management system pushes the policies out to each remote system . In this model, each remote system applies its own policies locally, it is just the policy information that is federated. Note: In this model you do not have centralized view/management of the records, just the polices.

2) Federating the application of retention etc. to the actual objects in the remote systems in place. This is the classic federated records management model that I discuss in this blog. I have just written a 2000 word overview of this use case as an article for one of the ECM magazines which I can send to you if you email me.

Use case #1 is typically less invasive that #2 and does give you a lot of benefits - it is not much easier to set up that #2 though! However, just knowing where the policies exist is useful; let's assume that a regulation changes and you have to find all objects in the enterprise that have that policy applied to it. Use case #1 would at least let you know where the policies exist even though you cannot actually manage it centrally.