Video Screencast Help

Archived emails with attachments - The item is no longer in the vault

Created: 15 Nov 2010 • Updated: 15 Nov 2010 | 11 comments

Hello,

We've got a problem where a number of users have attempted to un-vault email items with attachments and get the messages below:

 

Top of email: "There was an error loading this item -- some functionality may not be available"

Double click grey banner: "The item is no longer in the vault"

When viewing Archive Explorer: "The XML returned for the tree view is not as expected. (500)"

 

This only occurs for emails that were received and archived within a particular timeframe, approx September - October and initially thought it to be an isolated case when the first user logged the problem, however two others have now reported the same problems.

I'm concerned this is a widespread issue and we're unaware of the true extent.   The problem has been logged with enterprise support and they've said we need to identify the saveset ID for each affected email and recover the DVS file manually from backup if missing.   Once this has been done, we need to run EVSVR command to "Repair > RecreateMissingDBReferences" to re-add the SQL DB references.

The suggested solution could involve hundreds or thousands of mans hours and potentially over years to recover the data, which is obviously not practical and a great concern!   I don't like having to rely on backups to recover the emails either.

I've requested that the call be escalated for other suggestions or solutions.

Has anyone else had the problem or can suggest a fix?

Thanks in advance.

Mike

Symantec Enterprise Vault 2008 SP4 for Exchange

Discussion Filed Under:

Comments 11 CommentsJump to latest comment

WiTSend's picture

First I would run EVSVR to discover the extent of the missing items.  Knowing the extent of the problem would help in developing the appropriate solution.

Has there been any outages or problems with the storage platform? 

MikeS 2's picture

There have been no known outages around that period.

EVSVR has been run in report mode to attempt to discover all those affected items, however it did not discover anything.   It only did so once we restored some specific DVS files then ran the command again in repair mode with "Require Index Entries" unchecked (It wouldn't re-add the DB entries with this option checked nor did manually selecting 'Update Index Volume' on the user's archive).

Mike

JesusWept3's picture

This part concerns me

"EVSVR has been run in report mode to attempt to discover all those affected items, however it did not discover anything.   It only did so once we restored some specific DVS files then ran the command again in repair mode with "Require Index Entries" unchecked (It wouldn't re-add the DB entries with this option checked nor did manually selecting 'Update Index Volume' on the user's archive)."

To me this sounds like it's just a user has deleted their email

MikeS 2's picture

That's the original suggestion by Symantec when we believed it to be only one user.   Now two other users have the same problems and it affects dozens of emails within the same time window.

EVSVR was run in the mode below:

Operation: Verify

Option: Unknown (tried to extract from XML file)

 

I'm tempted to setup a new XML file and run EVSVR again.   Can anyone recommend what settings to use?   Also, does the vault and indexes have to be in read-only mode?  I'm hoping these commands aren't doing any damage or making the situation worse.

Mike

JesusWept3's picture

OK so my thinking is this
Back in EV2007 and prior, there were two utilities
one was DVSChecker and one was VSVerify and they performed two different tasks

1. VSVerify
VSVerify went through each SQL Entry and found the corresponding DVS File, if the DVS File didn't exist (meaning you had an orphaned entry in the database) you could then either remove the entry from the vault store database automatically OR you could find that item from back up and restore it to its original location

2. DVSChecker
DVSChecker scans the NTFS Vault Store Partition and for each DVS it finds, it will attempt to find a matching record in the vault store database, if the entry didn't exist in the database you could have it re-insert the record or you could delete the DVS file.

The thing that makes me believe its a delete is that you ran a report, it found nothing missing, so all the SQL entries matched a DVS file, but it ONLY reported something was wrong *after* you placed the DVS file back there so now it has a DVS file but no corresponding entry in the SQL Database, so now you're telling it to repair

BUT you can't repair because the item required it to be in the index first, which it didn't exist there, and then when you unchecked that it put it all back in.

So if theres no index entry, and theres no SQL entry and theres no DVS file.....its either a delete or an expiration. You may want to run an EVSVR check across that date range against the most exhaustive option, but it will take a long time.

And even then, if it says you are missing items, you will still have to get them from a backup and put them back, or you could choose to delete the entries, but they will not delete the shortcuts unless you have orphaned shortcut deletion switched on

MikeS 2's picture

Thanks for the response.

I really can't understand how this could have happened.  We don't have the Vault set to expire any data and I'm pretty sure three or more users haven't gone through and deleted the items (not sure how they'd delete the DVS files anyway).   The users have the shortcuts to the emails and can read the text, but can't open the attachments.

Not sure where to go with this one and am concerned about the potential scale of the problem..   Can you recommend what options to run in the report?   I'll have to run it on a Friday whilst the Vault and indexes are in read-only mode.

Mike

JesusWept3's picture

Well the thing is, if you go in to Archive Explorer or Search or Virtual Vault, you can delete items there, it won't delete the shortcut, and when you delete an item, it will delete the item out of the index, it will then delete the record for the item out of the saveset table.

After this it will be placed in the JournalDelete table of the vault store and then wait for 14 days by default (for the EV Dumpster) and then after that 14 days it will then go ahead and delete the DVS files and its corresponding parts.

After this the delete has completed, it will then keep that transaction for a further 32 days and then that will be it.

So first of all here's what i'd suggest

1. Open up the Vault Admin Console and go to Archives -> Exchange Mailboxes
2. Find a user that has reported the problem and go to the properties of the archive
3. Go to the Deleted Items tab, does it show that there are a number of items awaiting to be deleted?

4. On the General Tab it will show which Vault Store they belong to, note this down
i.e EVStore1

5. On the advanced tab you will notice they have an archive ID, copy the archive ID
i.e. 1C6D87EB5661D6940ACC8DE895024B2011110000EVSite.mydomain.com

6. Go to your SQL Server and open up a SQL Management Studio
7. Start a new query and run the following

USE yourVaultStore
SELECT ArchivePointID "Archive ID", ArchivedItems "# of Archived Items",
IndexDeletesPending "# of items awaiting Delete",
HighestIndexSeqNo "Highest Index #",
HighestPurgedJournalSeqNo "Highest Purged Index #"
FROM ArchivePoint
WHERE ArchivePointId = 1C6D87EB5661D6940ACC8DE895024B2011110000EVSite.mydomain.com'

This query will return how many items are in the archive and how many items exist in the index, if no deletes have been performed you should have a one to one ratio in most circumstnaces, however it may indicate that deletions have taken place

So for instance on my archive i have

Archive ID = 1C6D87EB5661D6940ACC8DE895024B2011110000EVSite.mydomain.com
# of Archived Items = 21352
# of items Awaiting Delete = 0
Highest Index Number = 21363
Highest Purged Index Number = 21329

What this tells me is i have 21,352 items currently indexed, though throughout the history of my archive i've actually had 21,363 items archive, so it shows that i've deleted 11 items, and it was fairly recently too as the last item deleted was an item with the seq number of 21,329 (so with in my last 50 items i've archived, i did delete something)

Note down the amount of Archived Items you have listed there
Next run the following query

SELECT COUNT(S.IdTransaction) "# of items archived"
FROM Saveset S, ArchivePoint AP
WHERE S.ArchivePointIdentity = AP.ArchivePointIdentity
  AND AP.ArchivePointId = '1C6D87EB5661D6940ACC8DE895024B2011110000EVSite.mydomain.com'

This will give you a count of how many items exist in the Vault Store saveset table for your particular user, this should equate to the number of archived items in the previous query,

Last but not least you should run the following query

USE yourVaultStore
SELECT * FROM JournalDelete

This will return all the items awaiting to be deleted or that have been deleted, you will be able to see the deletion dates, whether its been removed from the indexes, whether its been removed from disk etc etc etc

But lastly in regards to what EVSVR options to run

1. Open a command prompt
2. cd to your \Program Files\Enterprise Vault directory
3. Type "evsvr" and press enter
4. Type "edit" and press enter
5. In Operation, choose "Verify"
6. In Option, choose "Complete"
7. Select your date ranges from something like July to November
8. Hit Save As and choose a name such as "MissingSavesets.xml" and press OK
9. Press OK and it'll take you back to the EVSVR screen
10. type "start" and let it run

this will be probably the most thorough test you can run including checking the integrity of the savesets themselves
 

MikeS 2's picture

Thank you for your help so far JesusWept2!

Any ideas to how this could have happened?  I'm hoping it's only a small handful of users affected, but it's impossible to tell until a user reports a problem.

I'll work through the commands tomorrow, it'll be a huge task identifying and recovering all the DVS files, getting harder as time marches on.

For your last suggestion and running EVSVR.. Does the vault and indexes need to be in read-only mode?

Mike

JesusWept3's picture

Some other things to check
Open a command prompt and CD to where your W3SVC1 log files are
Then type 'find *.log "deleto2k.asp" > c:\deletes.txt"
This will then go through each IIS log you have and look for user deletes that have occured through deleteo2k.asp, that way you'll see what time the item was deleted, for which user, from what ip address etc

Also you may want to consider enabling auditing so that way you can make sure an item that doesn't exist genuinely has been deleted or not

As for EVSVR you can run that in either backup mode or in full read/write mode no problems, I you're concerned abouthe impact to server or archiving rates you can give the thread a lower priority

MikeS 2's picture

Your "find *.log..." command returned:

---------- U_EX100211.LOG

---------- U_EX100212.LOG

---------- U_EX100213.LOG

---------- U_EX100214.LOG etc

What would the command be to search the W3SVC1 log for a particular saveset ID to determine when it was deleted?

Will try the EVSVR command now and leave it running.   I suspect it'll take 24hrs+ like you said.

Mike

MikeS 2's picture

I've trawled the logs using the command:

find /c "SavesetID" *.log > c:\output.txt

From the output, I've gone to the first log identified and can see the "deleteo2k.asp" entry for each of the missing messages and it marries up around the time of the deletion.

Would this line in the log only appear if Archive Explorer was used or does it also record this entry if the user accessed the email via the COM+ toolbar?   Also, is it possible the user could have deleted the items accidentally?

Mike