Knowledge Base & Discussion Forum

[Feature][OBS/RPS] Replication through multiple TCP threads

Discuss technical questions on AhsayRPS

Moderator: Support Team

[Feature][OBS/RPS] Replication through multiple TCP threads

Postby intrikrakin » Tue Jan 13, 2009 7:27 am

Platform : Any
OS Version : Any
Product : OBS & RPS
Descriptions : OBS would connect to RPS and send replication traffic through multiple TCP threads.
Reasons : Replication currently is only performed using one TCP thread at a time
Workaround : non available

I have noticed that some of our East Coast clients are not pushing the amount of traffic to us that I would like to see. While trying to troubleshoot the bandwidth problems I have determined the issue is not so much with the speed of the line, but the fact that the OBS only makes ONE connection at a time to try and send a file.

I believe this is also a big reason that customers who have large (1TB-10TB+) accounts notice that the replication just never seems to catch up.

I think the benefits of increasing the amount of tcp threads the OBS uses to connect and send files to the RPS would be twofold. First, to double, triple, quadruple the bandwidth that the OBS uses for replication as it would allow it to send multiple files at the same time using multiple threads. Second, by being able to send multiple files at the same time the replication would sync much more quickly.

The reason I am making this thread is because when I submitted this request to Ahsay yesterday I got the standard reply of "we will look into it" response. If any other partners are familiar with me I made another enhancement request that is pretty important : viewtopic.php?t=165 : and Ahsay STILL has not implemented it after 2 years of being on their list of things to do.
I am making this thread on behalf of all the partners that can never get a good speed on replication or have given up on replication because it just never catches up to the OBS. Please let Ahsay know you want this feature and light a fire under their butts to get this implemented soon, not 2 years from now.
Backup & Replication server solutions
http://www.korcomputing.com/
intrikrakin
 
Posts: 803
Joined: Fri Aug 11, 2006 11:46 am
Location: Phoenix, Arizona

This would be great!

Postby kingdomcomputer » Sat Jan 17, 2009 9:58 am

We need this and need the replication to go faster than it does.
kingdomcomputer
 
Posts: 27
Joined: Sun Aug 27, 2006 10:48 am
Location: Lebanon, PA

Postby Doebe » Sat Jan 17, 2009 4:58 pm

the current RPS is a pain in the arse...

I am in!
It's not a bug, it's a feature!
Doebe
 
Posts: 102
Joined: Tue Sep 19, 2006 6:08 pm

Postby leemason » Sat Jan 17, 2009 5:52 pm

I would certainly agree that this is an issue and would be happy to see this pushed up the priority list.
BackupsAnywhere Ltd (http://www.backupsanywhere.com)
leemason
 
Posts: 670
Joined: Fri Aug 25, 2006 4:35 pm
Location: London, UK

Postby rrnworks » Mon Jan 19, 2009 1:11 pm

I would expect all Internet data transfers should be optimized for all modules - RPS, OBS, and OBM.

I'm curious is any work has been done at all in this area? For example, is Ahsay using techniques that are optimized beyond standard TCP/FTP?

For example:
www.rocketstream.com

Can Ahsay claim to offer performance similar to this?? Should they partner with rocketstream like they are already doing with storagecraft?
rrnworks
 
Posts: 14
Joined: Fri Jan 09, 2009 10:49 am
Location: Albuquerque, NM

Postby Nelsonn » Mon Jan 19, 2009 5:24 pm

The enhancement request of multiple thread for replication is already in our enhancement queue for some time. There is no given date for this feature request because there are a lot of uncertainties for this request, such as the impact of this change on OBS/RPS. We have scheduled a time in Q3 2009 to do a comprehensive feasibility study on this feature. Please note that we would not be able to provide any target date for this request until we have finished the comprehensive feasibility study.

intrikrakin wrote:The reason I am making this thread is because when I submitted this request to Ahsay yesterday I got the standard reply of "we will look into it" response. If any other partners are familiar with me I made another enhancement request that is pretty important : viewtopic.php?t=165 : and Ahsay STILL has not implemented it after 2 years of being on their list of things to do.

Please note that this feature is INCLUDED in v5530 with the use of AhsayRDR, please refer to the documentation: http://download.ahsay.com/temp/rdr/rdr-admin-guide.doc

We could understand that it takes a long period of time before their feature requests are available to them. This is because the priority of the requested feature is not high enough or the requested feature has a big impact on the product. Also, please understand on the fact that a feature may be more complicated than it looks. For example, the request of adding ability to point to multiple OBS servers on OBC side. From user point of view, it looks like that it only affects the client side, but actually we need to make some quite a lot of changes on the OBS side to support this request.

leemason wrote:I would certainly agree that this is an issue and would be happy to see this pushed up the priority list.

We have given the highest priority to this request already.

Thanks
Ahsay Online Backup
Nelsonn
 
Posts: 285
Joined: Mon Aug 07, 2006 12:12 pm

Postby rrnworks » Mon Jan 19, 2009 11:51 pm

Are multiple TCP connections part of OBM yet? Can this also please be added as an enhancement request.

I am surprised, since Internet transfer performance should be the top priority for an offsite backup solution like Ahsay?? Other backup vendors are already doing multiple tcp streams for the backup client.

In addition, I would be curious to know what Internet transfer protocol optimizations have already been done for OBS/OBM? Or are they just using plain FTP/HTTP??
Last edited by rrnworks on Tue Jan 20, 2009 6:55 am, edited 1 time in total.
rrnworks
 
Posts: 14
Joined: Fri Jan 09, 2009 10:49 am
Location: Albuquerque, NM

Postby intrikrakin » Tue Jan 20, 2009 1:35 am

Nelsonn wrote:We have scheduled a time in Q3 2009 to do a comprehensive feasibility study on this feature.
We have given the highest priority to this request already.

Q3 doesn't sound like highest priority at all.
Backup & Replication server solutions
http://www.korcomputing.com/
intrikrakin
 
Posts: 803
Joined: Fri Aug 11, 2006 11:46 am
Location: Phoenix, Arizona

Postby Sajid » Tue Jan 20, 2009 11:22 am

You do realize that the RPS module is for free right? And Ahsay is providing you with free support for RPS. Are you familar with the saying "you get what you pay for"? I think RPS module users got a very good deal already.

I think the benefits of increasing the amount of tcp threads the OBS uses to connect and send files to the RPS would be twofold. First, to double, triple, quadruple the bandwidth that the OBS uses for replication as it would allow it to send multiple files at the same time using multiple threads. Second, by being able to send multiple files at the same time the replication would sync much more quickly.

The reason I am making this thread is because when I submitted this request to Ahsay yesterday I got the standard reply of "we will look into it" response.


You seems to think that every request of yours is easy to implement, and that Ahsay should respond saying "Ya, this feature will be included in the next version real soon".

Software development does takes time and money, not every feature is important to you is as important to others. I made backup related enhancement request which is backup related features and benefits other partners, and they listen to us, the feature is now available on 5.5. RPS maybe a first priority for you, as you make most of your profit by selling your "RPS" service here. It may not be the first priority for other partners, who are actually in the online backup business rather than the "managed replication" business.
Black Star Services Company
Sajid
 
Posts: 38
Joined: Wed Aug 23, 2006 10:31 pm

Postby rrnworks » Tue Jan 20, 2009 12:28 pm

I believe the RPS module cost is built into the cost for OBS/OBM. Also, RPS is a key reason many of us have purchased OBS to begin with because it is such a great feature. That doesn't mean it doesn't have serious shortcomings that should be addressed as soon as possible. Lastly, I believe we are all just asking Ahsay to honor their word while listening to the providers who are making them successful.
rrnworks
 
Posts: 14
Joined: Fri Jan 09, 2009 10:49 am
Location: Albuquerque, NM

Postby intrikrakin » Tue Jan 20, 2009 2:54 pm

Sajid wrote:You do realize that the RPS module is for free right? And Ahsay is providing you with free support for RPS. Are you familar with the saying "you get what you pay for"? I think RPS module users got a very good deal already.

The RPS module is useless without an OBS module to send to it. The "free" doesn't really come into play.

Sajid wrote:You seems to think that every request of yours is easy to implement, and that Ahsay should respond saying "Ya, this feature will be included in the next version real soon".

Every feature request? I did a little digging and have found that every feature request I have ever made to Ahsay has actually never been implemented. That's not a very good track record.
viewtopic.php?t=308&highlight=
viewtopic.php?t=311&highlight=
viewtopic.php?t=2331&highlight=
viewtopic.php?t=2113&highlight=
viewtopic.php?t=165&highlight=

Sajid wrote:I made backup related enhancement request which is backup related features and benefits other partners, and they listen to us, the feature is now available on 5.5.

Great! Let me know what you are doing that I'm not doing to get my enhancements implemented. You seem to have better luck than me.


Sajid wrote:RPS maybe a first priority for you, as you make most of your profit by selling your "RPS" service here.

Not true. I am an Ahsay partner just like you. Sorry if you feel slighted somehow by my business model.

Sajid wrote:It may not be the first priority for other partners, who are actually in the online backup business rather than the "managed replication" business.

I'm actually in the business of "managed replication" and "offsite backup" and "managed OBS hardware" but who's counting.
Backup & Replication server solutions
http://www.korcomputing.com/
intrikrakin
 
Posts: 803
Joined: Fri Aug 11, 2006 11:46 am
Location: Phoenix, Arizona

Postby rrnworks » Wed Jan 21, 2009 12:49 pm

I would like to offer the following as to what can be done to increase file transfer performance, with both RPS AND OBM. Ahsay is currently doing at least one out of the four - not sure if they do on-the-fly compression? I am hoping they can add the other two asap, starting with multiple TCP streams and following with UDP optimizations. Obviously, other backup vendors are already doing this, so this is of major importance to stay competitive:

http://www.filecatalyst.com/products/faq.html#2

How does FileCatalyst accelerate file transfers?
FileCatalyst uses multiple techniques to yield exceptional results; in many cases, transfers are much faster than actual line speed. These techniques include:

FileCatalyst UDP-based protocol: FileCatalyst uses a patent-pending UDP-based protocol that is faster in transferring large data sets when there is high latency or packet loss over the network

On-the-fly Compression: FileCatalyst compresses the data it sends over the network in real time, meaning there is no time-consuming compression or decompression at the beginning or end of each transfer. File sizes may be reduced by 50% or more, creating the illusion of transfers that exceed line speed.

Multiple TCP Streaming: In situations where UDP is not possible, FileCatalyst can achieve some measure of acceleration by running transfers through multiple concurrent TCP streams, along with on-the-fly compression.

Delta Transfers: If a file has changed since its last transfer, FileCatalyst can send only the “deltas” (incremental differences) of the file, rather than resending it in its entirety.
rrnworks
 
Posts: 14
Joined: Fri Jan 09, 2009 10:49 am
Location: Albuquerque, NM

i aggree

Postby tstandage » Thu Jan 22, 2009 11:32 am

I have been using the replication server for about a month now. first of all, let me say this. i was very hopefull that ahsay had fixed all the problems. i had used it about a year ago, but it was way to buggy. since upgrading to 5.5 i had my hopes. this is my conclusion.

unless you have less then 2TB of data on at least a 100Mb network, this solution will NOT work.

it may work if you have 5 or 6 hundred 1GB accounts, but its risky.

right now we are replicating to a datacenter with a 10mb connection. We have a 155Mb connection up. As of 1 month now, it has only replicated 4 accounts at around 350GB.

internally, with our little servers, it has worked ok, execpt when it does its little sync thing, it completly fills up the log dir with logs and we run out of space. these servers have 80GB (for c drive) hard drives and they are completly out of room. this is very unnecessary.
Its hard to image that one of my servers has an 80Gb c drive, using 500GB for accounts and i dont have enough room on the c drive for logs (65GB of free space). so i cant use it without upgrading the c drive. sounds like an in expensive problem, but it is a pain in the neck to replace all of the drives on about 20 servers.

last, if this rps software is going to have any chance at working whatsoever, ahsay needs to do the following:

1. get rid of the stupid monthly or weekly sync that creates all the logs.
2. they need a copy thread for EACH drive letter/mount point. in other words, if you have 10 drive letters or partition with accounts, then there should be 10 threads copying/syncing data at the SAME time. then maybe it would catch up. it would be nice to choose how many threads are used as well. Also, we use mount points, RPS does NOT work on mount points. period. i starts at 1, goes to 10, 11, 12, 13, 14, 15 and goes back to 1. i brought this to ahsay's attention but they did nothing about it and closed the ticket as usual.

until this happens, RPS is worthless. robocopy is a mutch better solution. I should know, i sync almost 80TB of data every week. :x
tstandage
 
Posts: 111
Joined: Wed Nov 08, 2006 1:11 pm
Location: Mesa AZ

Postby Nelsonn » Fri Jan 23, 2009 5:36 pm

rrnworks wrote:Are multiple TCP connections part of OBM yet? Can this also please be added as an enhancement request.


The multiple TCP connections will be applied to all products.

rrnworks wrote:I am surprised, since Internet transfer performance should be the top priority for an offsite backup solution like Ahsay?? Other backup vendors are already doing multiple tcp streams for the backup client.


Let me clarify once again that we have already given the highest priority to this request. According to our development schedule, Q3 2009 would be the earliest possible date that we could start working on it.

intrikrakin wrote:In addition, I would be curious to know what Internet transfer protocol optimizations have already been done for OBS/OBM? Or are they just using plain FTP/HTTP??

Backup(OBM/OBS) are using standard HTTP/HTTPS and replication is using TCP sockets. In-file deltas and moved file detection are used to reduce the traffic between OBM and OBS. In OBSR v5530, there are some minor enhancement included to reduce the traffic between OBS and RPS, such as detecting the files that are being moved to the retention area and it would upload those files to RPS again.

intrikrakin wrote: Q3 doesn't sound like highest priority at all.

As I mentioned above, Q3 would be the earliest possible date that we could start working on this.

intrikrakin wrote: I did a little digging and have found that every feature request I have ever made to Ahsay has actually never been implemented. That's not a very good track record.


As I mentioned in another post, all requests are given a priority based on the importance and the feedback from customer in kb/forum. There are some other factors which might take into concerns on deciding the release date of a feature, such as the impact of adding that feature to the product, the time taken to adding a new feature. This is why sometimes a request with a lower priority might implemented first, rather than doing a higher priority feature requests. In general, we will schedule a request which involves lots of changes in coding in major/patch releases (e.g. v5210, v5510).

From your list of requests, it seems that most of them are still in the pending list except for viewtopic.php?t=165, that feature is already in the next coming release v5530.

rrnworks wrote:I would like to offer the following as to what can be done to increase file transfer performance, with both RPS AND OBM. Ahsay is currently doing at least one out of the four - not sure if they do on-the-fly compression? I am hoping they can add the other two asap, starting with multiple TCP streams and following with UDP optimizations. Obviously, other backup vendors are already doing this, so this is of major importance to stay competitive


We are using on-the-fly compression and multiple TCP streams would be something that we would like to implement asap. However, we do not have plan for using UDP protocol at this moment.
Ahsay Online Backup
Nelsonn
 
Posts: 285
Joined: Mon Aug 07, 2006 12:12 pm

Postby Sajid » Thu Jan 29, 2009 4:38 pm

intrikrakin wrote:The RPS module is useless without an OBS module to send to it. The "free" doesn't really come into play.

To my knowledge, you can "freely" use the RPS module if your customers have their own obs server, so "free" does come into play in your situation. All you need is your hardware investment, and the rps can be downloaded and used for free. Right?

Doing a little digging of your posts, you often promote your rps hosting service to partners at a charge? So you are profiting from free software and free support. Right? So, you've got a good deal. I personally would have nothing to complain about.

viewtopic.php?p=9129
viewtopic.php?p=8720
viewtopic.php?p=8716
viewtopic.php?p=6487
viewtopic.php?p=6239


intrikrakin wrote:Great! Let me know what you are doing that I'm not doing to get my enhancements implemented. You seem to have better luck than me.

Yes, I have submitted some enhancement suggestions to the support person by submitting tickets. The ones there were implemented are 256bit encryption and selective user replication. May be they are easier to implement i guess and or may be they are in demand :roll:
Black Star Services Company
Sajid
 
Posts: 38
Joined: Wed Aug 23, 2006 10:31 pm

Next

Return to AhsayRPS

Who is online

Users browsing this forum: No registered users and 3 guests

Looking for Rbackup Alternative | Vembu Alternative | Novastor Alternative | Asigra Alternative | BackupAgent Alternative? Try our product.


A wholly owned subsidiary of Ahsay Backup Software Development Company Limited  [HKEx Stock Code: 8290]