This is the first of a series of bulletins on important features of Exchange archiving, namely: single instance storage, retention management, PST management, mailbox size, multi-mailbox search, and item-level restore. Beginning with Exchange 2010, many small and medium-size organizations may find that sufficient features are included, so that third-party archiving tools are no longer necessary.

One of the lesser-known changes to Exchange 2010 is the removal of single instance storage (SIS). The reason for this is related to an architectural change, disk I/O performance, and the availability of cheap disk.

There tends to be a trade-off between better disk I/O performance and reduced storage capacity. Architecturally, Exchange 2010 introduces a new per-mailbox table structure that replaces the original per-database table structure. The original per-database table structure was optimized for SIS, but disk I/O suffered. The new per-mailbox table structure improves disk I/O, but without SIS.

In place of SIS, Exchange 2010 uses compression. Only large, redundant attachments files truly benefit from SIS; otherwise, compression delivers roughly the same volume of data as SIS.

For further information, see this Exchange team blog discussion.

… Bob Spurzem

In addition to his role as Ferris analyst, Bob is director of product marketing at Permabit, which offers a grid–based disk storage system.

Exchange 2010’s database strategy is very interesting. The new Database Availability Groups and the benefits they offer for data protection and quick recovery are striking.

Overall, Microsoft is optimistic that mailboxes will be able to grow to 10GB or more. However, it’s unclear how large mailboxes will perform in practice. Several concerns spring to mind:

• By leveraging low cost disk, it is feasible, according to Microsoft, to store email online for up to 10 years. If you assume a typical user stores 5 MB of new email daily, this translates to 1.8 GB of email per year. Allowing for the continued growth in average message size, this translates to perhaps 10GB mailbox sizes over three years. For 100 users, this means 1 TB of total storage for Exchange. For 1,000 users, 10 TB of email storage! Managing multiple terabytes of storage is no simple matter when you take into account disk failures, power consumption, disaster protection, daily maintenance and monitoring, and so on
• Database maintenance and ESEUTIL need space. Offline defragmentation requires 110% free disk space. Thus the amount of storage required should be doubled for maintenance purposes. True, Microsoft advises that offline defragging is unnecessary and inadvisable; hopefully this turns out to be wise counsel
• If we wish to use the new Database Availability Groups for data protection and failover, extra copies will be required. Two to three copies seems likely, which implies another 2-3X in email storage
• Single instance storage is no longer supported, which means message stores can’t be shrunk by 20% or 30% or so

In a typical scenario, for every 1 TB of primary database storage, perhaps another 3-4 TB of disk space may be necessary for various support capabilities, such as Database Availability Groups. This can quickly translate into a large amount of storage.

All in all, I doubt Microsoft’s claims that imply storing TB’s of email data online is a simple proposition.

… Bob Spurzem

I like all the Ferris webinars, but I am particularly interested in this week’s topic on Exchange 2010 archiving. The date and time is November 4th, 8:30am Pacific Time Zone. You can register on the Ferris.com website, here.

The reason I am looking forward to this webinar is the growing controversy that surrounds the new Exchange 2010 and its new archiving capabilities. This is another classic struggle between Microsoft and its third-party independent software vendors. If you believe the early Exchange 2010 messaging, then you may think that Exchange 2010 has full archiving built-in and the third-party products for archiving are not required. But is Microsoft telling us the complete story? Or are they tailoring their message to fit their needs?

What is the truth about full archiving capability and what features are lacking with Exchange 2010?

Good questions. And the answers can be found by attending the webinar.

… Bob Spurzem

(Editor’s note: Forgive the shameless self-promotion!)

The old dumpster in Exchange 2003 and Exchange 2007 was valuable for recovering deleted email, but it had some basic compliance shortcomings. (Want more information on dumpster? See here.) It only contained deleted email, not deleted calendars or contacts. It was not indexed or searchable, and users were allowed to purge email from the dumpster, perhaps in violation of email retention policy.

Exchange 2010 has some important improvements to the dumpster. It now:

• Includes all deleted items from the mailbox, including email, calendars, contacts, and more.
• Is indexed so it can be searched using the new multi-mailbox search tool in Exchange 2010.
• Is extended with new Purges and Versions folders:
  • The Purges folder keeps items that users purge from the dumpster for the length of the dumpster retention period.
  • The Versions folder keeps a copy-on-write snapshot of email that users modify.
  • The Exchange 2010 dumpster still operates with a retention period (14-day default) or custom. The new dumpster features are a welcome addition to Exchange.

By capturing all deleted items and preventing users from purging email, it closes two major “leaks” for email retention and compliance.

… Bob Spurzem

A growing number of businesses use Twitter for advertising. This recent Wall Street Journal article elaborates.

It is very easy to sign up for a Twitter account, and it costs nothing. Each post has a maximum of 140 characters, so the message is very brief. To find a person or brand name to follow, simply search for the name and then click the “follow” button.

I use Twitter to follow the SharePoint brand name. I receive information on events, product changes, and trends. I have found the content in Twitter to be useful in some cases; for example, a link to an interesting article or news about a upcoming event. But the majority of the information I receive is useless.

And therein lies the secret for using Twitter. Always strive to submit content that is meaningful and useful for readers. Who wants to read this real excerpt: “Friday – what do I want to focus on today?”

…Bob Spurzem

The next major version of Exchange, Exchange 2010, introduces a valuable new concept called a Database Availability Group (DAG). A DAG is group of up to 16 mailbox servers that use continuous replication to update database copies, communicate to manage failures that affect individual databases, and can provide automatic recovery from a variety of failures. A database copy is a single mailbox database with its own transaction logs. Storage Groups no longer exist in Exchange 2010. Exchange 2010 supports a maximum of 100 Mailbox Databases per server. Just like in the past, the size of each database is unlimited, but best practice is to limit size so recovery time is optimal.

DAGs are part of the Exchange 2010 high availability solution which now provides database-level failover. Each Mailbox Database can have up to 16 copies. Exchange will manage the replication of data automatically between the database copies to keep data consistent. Replication is based on the transaction log files, similar to how Exchange 2007 used logs files with LCR, SCR, and CCR, which by the way are discontinued in Exchange 2010. By having multiple copies of each database, traditional Exchange backup is no longer a necessity. Should a hardware failure occur, Exchange can simply mount one of the database copies. According to Microsoft, failover time is less than 30 seconds.

… Bob Spurzem

In one word, BIG. The 2009 SharePoint Conference attracted over 7,000 paying attendees, making it one of the largest Microsoft events of 2009. This is remarkable when you compare it to the 2009 Microsoft TechEd event, which drew about the same number of attendees and covers all Microsoft solutions. The buzz I heard on the show floor was that the event was “everything I expected and more” and that “the event should be offered twice a year.”

One reason for its success is the rapid climb of SharePoint as an enterprise content management solution. Organizations are rapidly deploying SharePoint to better manage Office files and this is one reason for the strong interest in how to deploy and manage SharePoint. Second is the strong partner ecosystem that supports SharePoint. Hundreds of partners participated in the event and shared their tips and strategies on improving SharePoint.

SharePoint will continue to drive large audiences. SharePoint is a semi-customizable solution that in some sense depends on the creative sharing of the community to achieve its full potential. It is this active exchange of information that will continue to make SharePoint gatherings a success.

… Bob Spurzem

Exchange 2010 offers some additional help for PST files in the form of a new Archive Mailbox. Archive Mailbox is an actual Exchange mailbox that is licensed, exists in the same Exchange database, and appears in Outlook alongside the primary mailbox. The seamless integration of Archive Mailbox with Exchange is done very nicely, and this will no doubt increase its acceptance.

Email content that exists in offline PST files is imported into the Archive Mailbox via a new PST archive tool that comes with Exchange 2010. The PST Import tool is basic in its initial release, but gets the job done. I recommend you allow time to manually organize PST files first because the PST Archive tool does not include a crawler to go out and find PSTs.

The advantage of moving PST data into the Archive Mailbox is to unclog your file share and desktops. Email content in the Archive Mailbox is centrally managed, protected, and searchable. This eliminates the risk of losing email content in existing PSTs.

If your organization is struggling with PST file overload–as many are–Exchange 2010’s Archive Mailbox will be a big help.

… Bob Spurzem

Employees at Boston’s City Hall are in big trouble with email according to this Boston Globe article. It seems the common practice is to delete email daily to reduce mailbox load. At least this is the reason provided by one employee. But who forgot about public record laws which require all city email to be saved for two years?

It’s hard to believe that a major city like Boston is not aware of its obligation to preserve email, but that seems to be the case. No matter; a state judge has issued an order for Boston to change its practice immediately. It was not mentioned what fines or penalties will be levied as a result of the lawsuit, but the violations can incur a fine of up to $500 or a prison sentence of up to one year.

Email archiving needs to be taken seriously by Boston and every other government agency in the United States.

…Bob Spurzem

On October 19-22, the annual Microsoft SharePoint conference will be held in Las Vegas. The event has sold out, with 7,000 tickets sold. This makes the event almost as large as the annual Microsoft TechEd show.

This year’s SharePoint show promises to be exciting for two reasons. First, we will learn about the next major release of SharePoint 2010 due in the first half of 2010. Second, we will have a chance to speak with all the SharePoint partners and see their new products and services.

I will be attending the show and will report back.

…Bob Spurzem

One of the interesting new concepts being introduced with Exchange 2010 is “Personal Archiving” vs. “Organizational Archiving.”

At the recent TechEd, a Microsoft instructor introduced personal archives as a means to reduce primary mailbox size and circumvent quotas without losing organizational control. In effect, personal archives are a new replacement for PST files.

Personal archives are part of Exchange Server and are associated with an existing primary mailbox. Email can be moved from a primary to an archive mailbox using policies. Users access both primary and archive mailboxes side-by-side via Outlook and OWA.

The main functions you get from a personal archive are:

• Basic message retention (Move-to-Archive Policy; Delete Policy; Hold Policy)
• No single instance storage
• Simple role-based access (e.g., so auditors can view user mailboxes)
• Basic keyword and metadata-based search across mailboxes, typically via browser
• Bulk PST import/export from file share; no PST crawler

Organizational archiving goes beyond the scope of personal archiving and delivers full mailbox capture for all users, full single-instance storage across all data, and advanced search and case management tools for e-discovery.

By way of comparison, a typical third-party email archival solution can be expected to deliver all or a portion of the following key functions:

• Logs, WORM, read only
• Single instancing/compression
• Configuration auditing
• Mailbox auditing
• Journaling metadata
• Rogue admin protection
• Regulatory accreditation
• Protected content (signing/encryption)
• Federated discovery, retention, and reporting across content
• Data mining and visualization
• Case management and advanced e-discovery
• Archive for Bloomberg data and other non-Microsoft IM data
• Monitoring and supervisory tools
• Archive for files and SharePoint

In effect, Microsoft is positioning the new archiving features in Exchange 2010 for personal archiving and leaving the door open for third-party solutions to deliver more advanced feature necessary for organizational archiving. Small organizations will find the basic features of Exchange 2010 satisfactory to reduce the strain on storage growth and eliminate PST files. But for organizations that require full email retention and advanced e-discovery, a third-party email archiving solution is the answer for the next few years.

… Bob Spurzem

If you are considering Exchange 2010 for archiving, be aware of two factors that can increase the cost of, and delay, implementation:

• The archiving features of Exchange 2010 require enterprise CALs. If you are not already using enterprise CALs, then you must pay the additional cost to upgrade.
• To use Exchange 2010 archiving, you must also upgrade to Office 2010. This will ship six to nine months AFTER Exchange 2010 ships. This will add a significant cost to an organization that has Office 2007 deployed and does not wish to upgrade all of its desktop machines. Without Office 2010, you can’t leverage the archive functionality.

… Bob Spurzem

Stories about lost email. This case involves the Governor of North Carolina, his wife, leading officials at North Carolina State University, and corruption.

From an IT perspective, the following points are salient:

• Although the matter is currently active, the email in question dates back to 2005. This raises the issue of email retention and accessibility. Are you able to retrieve email that is years old? In this case the IT staff was asked to retrieve email that was over four years old.
• IT does not get input into the matter. Legally, the email was asked for, and it is IT’s job to find it. IT needs to work with its organization’s policy committee on such issues as email retention, so there is clear understanding of how long email is to be saved and the cost to retrieve it.
• Should the email not be found, as is the situation in this case, then the judge can issue an adverse judgment. This basically tells the jury that they can assume the worst because key email has been obviously destroyed to protect those being accused. This is clearly bad for IT, and the entire organization.

Email archiving is a well-accepted solution for keeping email secure and accessible long term. It’s becoming more and more common, so the argument that one didn’t have the right technology in place is gradually being diminished. If your organization is still not sure if it needs to archive email, it may be worthwhile forwarding this article to upper management.

… Bob Spurzem

Exchange 2010 is the next version of Exchange Server. It is due for release at the end of 2009 or early 2010. Exchange 2010 includes many new features and enhancements for archiving.

It’s early days for Exchange built-in archiving, and storage management is an important area in which it falls short:

• It does not move archived email off the Exchange Server. Since email never leaves the Exchange Server, Exchange data protection, storage, and recoverability will all be adversely affected as the total Exchange storage increases.
• It (as well as all previous versions of Exchange) does not perform single instance storage across all of its stores. This means the email you archive in Exchange 2010 is not de-duplicated, which further compounds archive storage problems.

At last month’s TechEd, a Microsoft employee presented Exchange 2010 archiving and positioned the offering as a “personal archiving” solution and not an “organizational archiving” solution. This is correct. Today, an organizational archive must keep the archive data off-host, fully de-duplicated, and under full retention management.

Exchange 2010 archiving is a useful replacement for PST files and personal archiving. But it falls short of a true organizational archive solution. Today, only third-party solutions deliver all the features required.

… Bob Spurzem

If you have ever accidently deleted a document from SharePoint, then you know how difficult it is to restore a single item from SharePoint. The cause of this problem is that SharePoint has a relatively weak backup engine and it does not give you the option of performing single-file restores from a backup. SharePoint document versioning does not help either. When you delete a document, SharePoint Server removes all version history.

New solutions for SharePoint archiving can be the answer to this problem. SharePoint archiving solutions capture SharePoint content at the item level and keep it secure in a central repository for search and compliance. Individual items in the archive can also be copied back to SharePoint to, in effect, serve for single-item restore. So a SharePoint archival solution delivers benefits for both archival and data protection.

A single solution for SharePoint archiving and data protection improves SharePoint manageability and reduces cost.

… Bob Spurzem