1

I have a script written in Ruby that has maxed out a core in my server's Xeon processor for the last 2 hours. Since it's currently only using 1 of four possible cores, I want to try and rewrite the script to take advantage of all four cores.

I can use the .each_slice(n) method on the array that contains my data, but I'm curious as to what would then be the best/most efficient way to then write this data to a file. It seems that I have a couple options.

  1. Pass the file object to the functions being called by the Thread.new function (I assume this is legal in ruby?) and have them write as they see fit.

  2. Have each function store the results in arrays, return the array on completion and let the main program then write to the disk.

  3. Pass the same array object to each item and have them all add to it, then sort it.

My assumption is that although 2 would probably be more memory intensive, it'll by far be most efficient method. Is there a different way to accomplish what I'm doing?

  • 1
    Note that the current MRI Ruby uses a global lock that means only one thread can be active at once, so you can’t use multiple cores concurrently like this. You may be able to use JRuby or Rubinius which don’t have this limit, or you may be able to use multiple processes (although that would be more difficult to do). – matt Sep 8 '13 at 15:11
2

You didn't mention if it matters what order the output is in. If it really doesn't matter I'd use a single-threaded Queue to receive messages from multiple reader/processor threads. Then another thread could be in charge of reading from the Queue and writing to the output file.

If order matters then your #2 seems a good idea.

I am concerned that with your suggestions #1 and #3 that if one is not very careful with the locking around the shared output object (file pointer or array) that one would be in 'undefined behavior' territory.

  • I concur regarding the risk of having parallel writes to a single write destination- this part would need to be locked somehow. – glenatron Oct 8 '13 at 13:29
1
  1. Pass the file object to the functions being called by the Thread.new function (I assume this is legal in ruby?) and have them write as they see fit.

The way you describe it each thread will process the whole file.

  1. Have each function store the results in arrays, return the array on completion and let the main program then write to the disk.

This is possible, however it may be memory intensive to fire up a new Thread.

  1. Pass the same array object to each item and have them all add to it, then sort it.

Again this is possible, but may suffer from the same issues as 2.

If the tool maxed one core it may use a good chunk of 4 cores and starting up too many threads will use capacity you may want for the processing of the file. You may want to use a thread pool and farm out the work from there.

You may also want to preprocess the file, partially processing it or splitting it into more manageable chunks from processing.

  • I didn't mean to pass the file being read, rather the object being written to. – Jacobm001 Aug 16 '13 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.