8

There is an utility we have which is used to upload files (and perform other operations on the file) to a network shared location.
The file size tends to vary from a few mb to 500 mb.
A suggestion has come up that we should maybe support multi-threading when uploading the files to the shared location - not required to do it in byte chunks - each thread should pick one file and try to upload.

I am not so sure if multithreading can speed up IO operations like this. Is my hunch valid?

If indeed we are required to build this functionality I was wondering what would be a good design approach for the copy file engine?
Would it make sense to use a tool like robocopy (I read the newer versions support multithreading)?

Edit: Apologies for the delay and missing some vital information.
This utility is built using C# (.Net 2.0) and any future update also has to be using .Net (framework version is not a constraint). The utility is installed on the machines of the users (around 20 all on WinXP). The target share is on Win2k3 server.

Edit 2: have decided to run some tests with a simple application implementing the file upload through TPL. Post this analysis we will decide whether to go ahead or not. Thanks everyone for the help extended.

Improve this question
3
  • 1
    What programming language? In C, a more idiomatic approach might be to use asynchronous I/O, using a select loop instead of threads. Although doing so requires you to "turn your code inside-out" (the code to copy a file isn't a straightforward sequence of commands anymore), you won't have to worry about thread synchronization.
    – Joey Adams
    Dec 2, 2011 at 23:43
  • Probably the easiest reasonable solution is to let the OS handle it all: SHFileOperation(FO_COPY). That gets you all optimizations that the people at Microsoft considered reasonable.
    – MSalters
    Dec 30, 2011 at 13:15
  • Cough robocopy cough... you could automate it with something like robomojo
    – James Snell
    Jul 9, 2013 at 10:40

9 Answers 9

Reset to default
19

This depends on what the limiting factor is, doesnt it? If the bottleneck is the utility program, then sure, running more than one copy or using more threads will speed things up. If the network is the limiting factor, then adding multiple instances of the utility isnt going to help since you still will be stuck moving at most X bytes per second. In fact it might hurt because you have the additional overhead of a second copy of the app. Same with disk-IO. You can only copy as fast as either machine can read from and write to disk. If thats already max'd out, adding copies isnt going to help.

What you need to do is test to see what the bottleneck is, and go from there.

Improve this answer
11

How multithreading will not help:

Multiple threads simultaneously reading from the client's disk or simultaneously sending stuff over the network will not help at all, since there most likely exists only one path of communication between the client and the server, the client is most likely reading files off from a single hard-drive, and the files are most likely being written on a single hard-drive on the server. (Even if the server has RAID, it will make some difference, but not much.) On the contrary, as it has already been pointed out, performance will probably degrade, because there will be constant seeking between the files that are being read in parallel on the client, and constant seeking between the files that are being written in parallel on the server. Also, the files may end up being stored badly fragmented on the server.

How multithreading will help:

However, multithreading can help in a different way: with just two threads on the client, the file I/O can be desynchronized from the network I/O. This means that the client can be simultaneously transmitting a chunk of a file while reading the next chunk from its disk. (The server is already capable of simultaneously writing a chunk of a file on the disk while receiving the next chunk from the network.) This would greatly speed up the transfer process, because the client would tend to keep either the network channel or the disk channel (whichever is slower) saturated, instead of accessing each one at a time, intermittently. I would guess that every specialized file copy utility out there should be smart enough to do that, but I may be wrong, so if "Robocopy" advertises that they do multi-threaded copy, that's fine, go with that.

EDIT: I corrected the bit I had written about RAID.

EDIT: I corrected the bit about requiring two threads on the server.

I guess the most important thing here, (as it is almost everywhere,) is measurement. You don't have any control over how these utilities work, so you will only know whether you are doing it the fastest way possible if you measure the throughput to see if it is near the advertised throughput of your disk or network (whichever is smaller.)

Improve this answer
3
  • This seems like a great idea, but I don't understand how file transfer utility can actually do this unless it has remote execution privileges (or the transfer protocol somehow inherently supports this). I'm trying to write a utility to do regular transfers of a gigantic number of files from a NAS, and I really need to find ways to cut down time.
    – Asad Saeeduddin
    May 14, 2015 at 14:22
  • Your question is very perceptive, and it points to an inaccuracy in my answer. The server should already be desynchronizing network I/O from disk access, by virtue of how servers usually work: they tend to do asynchronous I/O instead of spawning a separate thread for each client which can only be doing one thing at a time. So, you should not need to worry about the server, only about the client. I will reword my answer.
    – Mike Nakis
    May 14, 2015 at 15:35
  • @Asad Also, please note that by "server" in this case I mean any machine which is on the receiving end of the transfer. It does not need to be a machine with a designated server role.
    – Mike Nakis
    May 14, 2015 at 20:45
9

While copying a lot of smaller files, multithreading can help because there tend to be gaps in the data transfer while the program is searching directories for the next file, opening it and getting the data.

Multithreading will also help when the client and server both have parallel data storage such as RAID or SSD: anything that performs better with higher queue depth numbers.

Other than that, it will often slow things down. For example, making a single hard drive read or write two files at the same time will force it to repeatedly seek from file 1 to file 2.

Improve this answer
2

I work for Data Expedition, Inc. which, as Emmad mentioned, produces commercial software for this type of scenario. Multithreaded file transfer can have benefits, but you have to carefully understand what your performance bottlenecks are.

Any network path will have at least dozens of hardware and software components that the data has to pass through. The slowest of all of them will determine your speed. But how you move the data will change how those components behave.

Lots of background on that here: http://www.DataExpedition.com/support/notes/tn0009.html

Running parallel TCP's can help when individual TCP speeds are falling far below the capacities of the network, the disk, and the CPU.

But if you're looking at network speeds of more than tens of megabits per second, then parallel data transfers will exponentially reduce your disk I/O due to thrashing the hard-drive. It can quickly fall to the point where disk access becomes much slower than the network capacity. Choosing the right read/write block size can help, but that will depend on the particular hardware. Also keep in mind that Windows XP/2003 has very limited paged-pool memory, which can make it unstable if speeds go over about 200 megabits per second.

On the flip side, if the network is slower than a few tens of megabits per second, then running many parallel TCP's can push the latency up to the point where individual sessions begin to slow down or even drop their connections. Again, its a matter of experimentation to find what level of parallelism will work for any given path and conditions.

So, multithreaded file copy can help if you have a known data path and can take the time to fine tune the number of parallel sessions and your disk I/O. But it does require that you retune whenever conditions change, and can be disruptive if you overdo it. That's why we have chosen to avoid parallel transfers in our own software, just as we avoid TCP.

Improve this answer
1

In addition to what has been said, consider: - There must be a task on the client to create the chunks and another on the server to put them back together as 1 file again. This requires some work.

  • A good thing about small chunks is that you can re-send pieces of a file if the process fails instead of sending the big file all over.

  • Consider requesting a 'larger pipe' between your client and the server.

  • Consider Zipping the big file before sending it (I am not sure if this would help in case of multi media files since they are sometimes are already compressed).

  • Consider using a commercial file transfer utility such as:

DataExp

Improve this answer
0

If you are talking about one big file, multithreading won't really help. You're going to be I/O bound, so using a single thread won't slow THAT upload down.

What you do may to worry about, though, is resource contention (assuming you are writing the server, too). If you're handling the upload in the thread that also accepts and processes new requests, other requests will be waiting. As long as you defer back to the selector queue after reading a chunk from the socket and writing it to disk, though, you should be fine.

Improve this answer
0

Doing what you suggest in a naive fashion will kill your throughput, the choke point is disk I/O and not getting files ready.

I'll suggest using one thread that receives files to work with and queues them up for the copy and then keeps a sequential copy going on anything in the queue; your supplier thread is responsible for getting the files read to queue up. In this way you're not thrashing the file system on the shared drive(s) and you're not doing files one at a time with gaps to prepare the next, you're preparing and sending simultaneously.

Bonus is that there's only one synchronization point at the queue to worry about.

Improve this answer
0

Instead of implementing parallel upload yourself, you might consider existing protocols and tools. E.g. the ftp protocol and the lftp tool (lftp can transfer several files in parallel).

So it is probably much easier and more robust to use lftp scripts or control lftp from your application instead of implementing everything from scratch.

Improve this answer
0

It all depends on where the limiting factor is.

Multithreading could help if there are round trip delays or other gaps in transmission, and the threads help fill in the gaps.

Multithreading could hurt if it has the effect of causing your disk to rattle back and forth, trying to keep all the threads supplied with data.

etc.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.