Knowledge Base & Discussion Forum

Syncing with Robocopy

Discuss technical questions on AhsayRPS

Moderator: Support Team

Syncing with Robocopy

Postby choldsworth » Fri Aug 15, 2014 11:24 pm

We have about 16TB of data on our OBS server that we'd like to sync to another array. I've attempted to use RPS, but unfortunately it runs so slowly that using it as a replication solution is just not possible.

I've decided to use Robocopy to sync my data, but I've run into the issue of locked files. Either, Robocopy locks the files and messes up my backups, or the backup is running and Robocopy isn't able to copy it.

I know others are using Robocopy, rsync, and other solutions to replicate their data, so my question is, how are you getting around this? And those of you that have, with Robocopy, would you mind sharing your script with me?

Any help is appreciated.

Thanks.
choldsworth
 
Posts: 3
Joined: Fri Aug 15, 2014 11:19 pm

Re: Syncing with Robocopy

Postby PG » Sun Aug 17, 2014 7:31 am

I use both rsync and robocopy, nether of which im really happy with.

We have all our storage broken up into about 5TB user homes

Rsync is faster, but I have found issues with it on Windows where it does not always purge out the files at the destination end.

Robocopy is quite slow, a pass-over a 5TB can take up to 10 days.

This is the robocopy script I wrote, basically it runs a continuous loop, once it completes it calls itself to run again outputting an entry to a log file at the time it starts and completes.

This is written to run on the destination server and be called robocopy-sv1-vol1.bat

@echo off
TITLE robocopy-sv1-vol1

echo "Time Started %date% %time%" >>sv1-vol1-robocopy.log
ROBOCOPY /MT:4 /E /PURGE /XO /W:0 /R:0 \\HOST1\sv1-vol1-live "E:\OBSHOMEl"
echo "Time Completed %date% %time%" >>sv1-vol1-robocopy.log

call robocopy-sv1-vol1.bat


the rsync one is very similar, running under rsync for cygwin

echo "Time Started storage %date% %time%" >>rsync-sv2-vol1.log
rsync -avr --delete --stats --progress --exclude '*.tmp' 'rsync://HOST2/sv2-vol1/storage' '/cygdrive/i/STORAGE'
echo "Time Completed storage %date% %time%" >>rsync-sv2-vol1.log

call rsync-sv2-vol1.bat

In saying that I haven't used RPS for many years I am re-trialling it with one of our servers where robocopy takes a considerable amount of time to complete, thus far it is significantly slower, so I doubt it will hold any ground.
PG
 
Posts: 1023
Joined: Thu Feb 18, 2010 5:57 am

Re: Syncing with Robocopy

Postby choldsworth » Tue Aug 19, 2014 12:25 am

Do you not have issues with locked files? I've noticed in testing with my own script that I've had issues with Robocopy having to skip files because Ahsay has them locked, as well as vice versa where Ahsay backups screw up because robocopy has something locked.

I've been running a script that basically goes through all of the backupset folders individually and kicks off a robocopy instance.

Here is an example of the robocopy portion:

robocopy R:\iks T:\iks /MIR /MT:8 /B /NP /NS /NC /NFL /NDL /R:0 /W:0 /copy:DAT /LOG+:c:\logs\iks_seed.txt
timeout /T 5
exit

Now, my backupsets are configured with advanced retention periods - a Weekly one that keeps 4 snapshots, and a Monthly that keeps 12. Backups occur daily and are incrementals, except for every month it takes a differential.

Since I am using an advanced retention policy, I'm not able to do Delta merging, which has resulted in a lot more files in OBS, and slower recovery times. I'd like to start doing delta merging to assist with faster recovery, however, I've read that RPS doesn't really play well with delta merging, and it would ultimately result in additional overhead for replication. Anyone experience with that?

I really just want a way to do replication that works, and does so in a timely fashion. I've spent so much time on this that quite frankly I'm wondering if another backup solution might be the answer.
choldsworth
 
Posts: 3
Joined: Fri Aug 15, 2014 11:19 pm

Re: Syncing with Robocopy

Postby PG » Tue Aug 19, 2014 4:40 am

choldsworth wrote:Do you not have issues with locked files? I've noticed in testing with my own script that I've had issues with Robocopy having to skip files because Ahsay has them locked, as well as vice versa where Ahsay backups screw up because robocopy has something locked.

I've been running a script that basically goes through all of the backupset folders individually and kicks off a robocopy instance.

Here is an example of the robocopy portion:

robocopy R:\iks T:\iks /MIR /MT:8 /B /NP /NS /NC /NFL /NDL /R:0 /W:0 /copy:DAT /LOG+:c:\logs\iks_seed.txt
timeout /T 5
exit

Now, my backupsets are configured with advanced retention periods - a Weekly one that keeps 4 snapshots, and a Monthly that keeps 12. Backups occur daily and are incrementals, except for every month it takes a differential.

Since I am using an advanced retention policy, I'm not able to do Delta merging, which has resulted in a lot more files in OBS, and slower recovery times. I'd like to start doing delta merging to assist with faster recovery, however, I've read that RPS doesn't really play well with delta merging, and it would ultimately result in additional overhead for replication. Anyone experience with that?

I really just want a way to do replication that works, and does so in a timely fashion. I've spent so much time on this that quite frankly I'm wondering if another backup solution might be the answer.



Yes I come across locked files occasionally, the /W:0 /R:0 basically means 'chuck it in the f__k-it bucket & move on' and hope its not locked on the next-pass
PG
 
Posts: 1023
Joined: Thu Feb 18, 2010 5:57 am


Return to AhsayRPS

Who is online

Users browsing this forum: No registered users and 1 guest

Looking for Rbackup Alternative | Vembu Alternative | Novastor Alternative | Asigra Alternative | BackupAgent Alternative? Try our product.


A wholly owned subsidiary of Ahsay Backup Software Development Company Limited  [HKEx Stock Code: 8290]