Knowledge Base & Discussion Forum

Index corruption

Discuss technical questions on AhsayCBS release

Moderator: Support Team

Index corruption

Postby JasonEde » Thu Jan 07, 2016 1:18 am

Since a server with OBM crashed then I've had "error interrupted" in the logs.

After examing the logs in detail I noticed this.

[erro] [1449653437242] Cannot get remote file list ([g] [BlockFile.commitFromJournal] Failed to commit journal, caused by [RuntimeException] [CommitJournal.run] error, caused by [f] [JournalFile.parseJournalEntry] Invalid type=126, raf: C:\Users\user\temp\1449653392978\OBS@1449653437242\index\index-s0.j01, lOffset: 42602434)

Running a rebuild/CRC check via the client said that the journal was corrupt and the only solution was to delete all of the backup set files and re-upload all again.

Is this the only way to recover if a server crashes whilst running a backup and corrupts the index? That seems a tad excessive for a simple index corruption. Surely the server keeps backups of this and should be able to recover if it detects there is a problem. Even if it has to upload a few days worth of changes it's better than the entire backup set!
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Re: Index corruption

Postby Support2 » Thu Jan 07, 2016 3:23 pm

Hello Jason,

If the machine running AhsayOBM v7.5 crashed then this may cause the local copy of the index files to corrupt. IT does not affect the indexes on CBS or other related storage destinations.

To resolve this issue try:

1. Stopping all backups/restore jobs
2. Stop AhsayOBM service.
3. Exit AhsayOBM
4. Delete the contents from the following folder C:\Users\user\temp\1449653392978\OBS@1449653437242

When you restart the backup job AhsayOBM will re-sync the latest copy of the index files from the storage destination to the local AhsayOBM machine again.

To address your question about recovering from corrupt indexes on AhsayCBS. As a best practice we recommend you:

1. use our replication module this will ensure you have an additional copy of your customer data
2. ensure each backup set is configured to backup to more than one storage destination

These measures will provide your customers with maximum data protection .
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: Index corruption

Postby JasonEde » Thu Jan 07, 2016 4:18 pm

Hi Support,

Can this information be added to the knowledgebase as I think it will affect others as well.
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Re: Index corruption

Postby Kash » Thu Feb 04, 2016 10:25 pm

Hi,
I have just SEEDED 1.7TBs of data and getting same issue.
I have go through troubleshooting process but I am still getting same error message.

Can you help.

Error below

Index file for destination "CBS" of backup set "set name" is found to be corrupted. It is required to delete all data in this destination to fix this problem. Do you want to do so? (Error="[g][BlockFile.init] Missing BDB file while journal file exist: C:\Users\Administrator\temp\1453993589482\OBS@1454575132880\index\index-s0.j7d")
Kash
 
Posts: 11
Joined: Thu Feb 04, 2016 5:01 pm

Re: Index corruption

Postby Support2 » Fri Feb 05, 2016 5:47 pm

Kash wrote:Hi,
I have just SEEDED 1.7TBs of data and getting same issue.
I have go through troubleshooting process but I am still getting same error message.

Can you help.

Error below

Index file for destination "CBS" of backup set "set name" is found to be corrupted. It is required to delete all data in this destination to fix this problem. Do you want to do so? (Error="[g][BlockFile.init] Missing BDB file while journal file exist: C:\Users\Administrator\temp\1453993589482\OBS@1454575132880\index\index-s0.j7d")


I have already addressed your query in this thread https://forum.ahsay.com/viewtopic.php?f=170&t=14609
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: Index corruption

Postby cglmicro » Sun Mar 13, 2016 4:00 am

I would add this:
If you can't delete the folder (i.e.: files inside are in use) even if you stopped the AHSAY service, and you've killed the notification tray icon, it's possible that a process is still running: invoke TASKMGR.EXE and kill the process bJW.EXE.
cglmicro
 
Posts: 53
Joined: Fri Feb 27, 2015 9:28 am

Re: Index corruption

Postby SINC_dmack » Sat Apr 30, 2016 1:38 am

Per my comments in another thread, I've found that index errors are only reliably fixed if the backup set is completely deleted (not just the backup data), and then recreated and reseeded. If you're going to go through the trouble of reseeding 1.7TB, recreate the backup set completely to minimize the chance of having index problems in the future.
SINC_dmack
 
Posts: 77
Joined: Thu Mar 31, 2011 2:57 am

Re: Index corruption

Postby JasonEde » Thu May 05, 2016 8:52 pm

We now have 2 servers, that after running backups fine for weeks have suddenly started with errors similar to those above.

[BlockFile.commitFromJournal] Failed to commit journal, caused by [RuntimeException] [CommitJournal.run] error, caused by [f] [JournalFile.parseJournalEntry] Invalid type=16, raf: G:\Backup Temp Folder\1460636032192\OBS@1460636146134\index\index-s0.j00, lOffset: 8108425)
[2016/05/05 13:42:50] [cbs] [1460636146134] erro,"Cannot get remote file list ([g] [BlockFile.commitFromJournal] Failed to commit journal, caused by [RuntimeException] [CommitJournal.run] error, caused by [f] [JournalFile.parseJournalEntry] Invalid type=16, raf: G:\\Backup Temp Folder\\1460636032192\\OBS@1460636146134\\index\\index-s0.j00, lOffset: 8108425)",0,0,0,,

When we try and use the client to rebuild the indexes we get told that the CBS server index is corrupt and all backup data needs to be deleted and re-seeded. However, with nearly a TB of data that isn't really an option.

We've followed the instructions above for clearing out local caches and have told Ahsay this, but are still getting told to repeat the same process over and over again. It is looking like the indexes on the server have somehow got corrupted. Is there a way of rebuilding these indexes?
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Re: Index corruption

Postby Support2 » Sun May 08, 2016 3:57 pm

Hello Jason.

The following KB article may be of help:

ISSUE: Index file for destination of backup set is found to be corrupted (User prompted to delete all data when performing data integrity check) (5153) https://forum.ahsay.com/viewtopic.php?f=206&t=14853
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: Index corruption

Postby JasonEde » Mon May 09, 2016 4:26 pm

Hi,

We will try this. Is there not a way of rebuilding the index server side without having to sacrifice a days backup? At the very least couldn't the server keep a backup copy of the indexes for each set that is indedependently calculated and checksummed?
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Re: Index corruption

Postby Support2 » Tue May 10, 2016 4:12 pm

Hello Jason,

We have fine tuned the logic of CRC check in v7.7.0.0 to reduce the probability of deleting all backup data when the index is found to be corrupted during data integrity check by making use of other available index files on the backup set.

We expect the roll out of AhsayCBS v7.7.0.0 stable release to begin within 24 hours. Please subscribe to one of our social media channels to receive updates and notifications on new products releases:

https://twitter.com/ahsay
https://www.facebook.com/ahsaybackup
https://plus.google.com/+AhsayBackup
https://www.linkedin.com/company/ahsay- ... on-limited
https://www.ahsay.com/rss/technical-updates.rss
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: Index corruption

Postby JasonEde » Wed May 11, 2016 7:59 pm

I've tried the KB referenced. We had to go quite a way back through the previous backups to find a index that wasn't corrupted which implies that it was corrupt for a backup or 2 before it started failing on the client.

I've looked at around the time it started failing and it seems to have occured just after another backup set was added. At that point the extra backup set was added as the admin user we normally use on the server and not the backup user that was used to add the original backup. This added the extra user to the home.txt file and it looks like there might have been a permissions issue to the temp folder that by default resides within the user profile. could that have caused the index corruption we've observed? I'll be moving the temp folder elsewhere as we normally did that with obs6 as well.
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Re: Index corruption

Postby JasonEde » Thu May 12, 2016 5:35 pm

We're now seeing the below error on quite a few instances running on 7.7

1463045529641,2016-05-12-10-32-09,ERROR,Utilities,"Please run integrity check later for destination \"1450200446717\". Error: \"[d] [ObsManager.delete] failed in delete file, path: 1450200386527/blocks/2016-05-10-19-00-00/0/000000.bak, http code: 500, <html><head><title>Apache Tomcat/7.0.59 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.NullPointerException\n com.ahsay.afc.db.bdb.c.e(Unknown Source)\n com.ahsay.afc.db.bdb.c.a(Unknown Source)\n com.ahsay.afc.cloud.local.e.c(Unknown Source)\n com.ahsay.afc.cloud.local.e.m(Unknown Source)\n com.ahsay.afc.cloud.B.h(Unknown Source)\n com.ahsay.obs.obc7.FileService.delete(Unknown Source)\n com.ahsay.cbs.aK.doDelete(Unknown Source)\n sun.reflect.GeneratedMethodAccessor119.invoke(Unknown Source)\n sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n java.lang.reflect.Method.invoke(Unknown Source)\n com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)\n com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)\n javax.servlet.http.HttpServlet.service(HttpServlet.java:727)\n org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n com.ahsay.obs.www.EncodingFilter.doFilter(Unknown Source)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.59 logs.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/7.0.59</h3></body></html>\n, caused by [n] <html><head><title>Apache Tomcat/7.0.59 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size=\"1\" noshade=\"noshade\"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.NullPointerException\n com.ahsay.afc.db.bdb.c.e(Unknown Source)\n com.ahsay.afc.db.bdb.c.a(Unknown Source)\n com.ahsay.afc.cloud.local.e.c(Unknown Source)\n com.ahsay.afc.cloud.local.e.m(Unknown Source)\n com.ahsay.afc.cloud.B.h(Unknown Source)\n com.ahsay.obs.obc7.FileService.delete(Unknown Source)\n com.ahsay.cbs.aK.doDelete(Unknown Source)\n sun.reflect.GeneratedMethodAccessor119.invoke(Unknown Source)\n sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n java.lang.reflect.Method.invoke(Unknown Source)\n com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)\n com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)\n javax.servlet.http.HttpServlet.service(HttpServlet.java:727)\n org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n com.ahsay.obs.www.EncodingFilter.doFilter(Unknown Source)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.59 logs.</u></p><HR size=\"1\" noshade=\"noshade\"><h3>Apache Tomcat/7.0.59</h3></body></html>\""
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Re: Index corruption

Postby dexter » Fri May 13, 2016 3:02 pm

I upgraded to version 7.7 from 7.5 yesterday and overnight 50% of the OBS auto updated to 7.7.

This morning I can see that all 7.7 obs jobs failed with an error similar to the error mentioned in the above post.

Failed to upload cached index file C:\Users\emadmin\temp\1451133840556\OBS@1451133972083\index back to 1451133840556/blocks on OBS()
, Reason = "[ObsManager.delete] failed in delete file, path: 1451133840556/blocks/index-s0.j00.100.154a132656f.cgz, http code: 500, <ht
ml><head><title>Apache Tomcat/7.0.59 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-co
lor:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font... (truncated)

Can Ahsay support please let me know if this is a problem to my data and how to correct it? What does it mean?

Dexter
dexter
 
Posts: 22
Joined: Tue Jan 26, 2016 8:22 pm

Re: Index corruption

Postby JasonEde » Tue May 24, 2016 4:44 pm

We've heard from Ahsay. It seems like there is a case where the index files can be locked and therefore cannot be updated (i.e. cannot be uploaded at end of a backup job) or files cannot be deleted (index integrity check fails). They're working on a fix.
JasonEde
 
Posts: 274
Joined: Mon Oct 08, 2007 1:41 am
Location: England

Next

Return to AhsayCBS v7

Who is online

Users browsing this forum: No registered users and 1 guest

Looking for Rbackup Alternative | Vembu Alternative | Novastor Alternative | Asigra Alternative | BackupAgent Alternative? Try our product.


A wholly owned subsidiary of Ahsay Backup Software Development Company Limited  [HKEx Stock Code: 8290]