Imagine a scenario as follows: Lets say you have a central computer which generates a lot of data. This data must go through some processing, which unfortunately takes longer than to generate. In order for the processing to catch up with real time, we plug in more slave computers.

Further, we must take into account the possibility of slaves dropping out of the network mid-job as well as additional slaves being added. The central computer should ensure that all jobs are finished to its satisfaction, and that jobs dropped by a slave are retasked to another.

The main question is: What approach should I use to achieve this?

But perhaps the following would help me arrive at an answer: Is there a name or design pattern to what I am trying to do?

What domain of knowledge do I need to achieve the goal of getting these computers to talk to each other? (eg. will a database, which I have some knowledge of, be enough or will this involve sockets, which I have yet to have knowledge of?)

Are there any examples of such a system? The main question is a bit general so it would be good to have a starting point/reference point.

Note I am assuming constraints of c++ and windows so solutions pointing in that direction would be appreciated.

  • 1
    Look for queue or workflow systems, examples: aws.amazon.com/sqs en.wikipedia.org/wiki/Message_queue etcetera. The queue system can take care of distribution, checking and re-assigning. – Luc Franken Jan 2 '13 at 11:03
  • 2
    The term you're looking for is parallelization. Before you head down this road, make sure your problem can actually be solved in parallel (e.g., the result for item n doesn't depend on the result for item n-1). – Blrfl Jan 2 '13 at 11:27
  • Look at boinc – user40980 Jan 2 '13 at 14:30
  • @Blrfl I think parallelization is not usually associated with this, parallelization is usually associated with multi-threaded or multi-process execution on a single machine using multiple cores/processors or concurrent execution on the same machine. Distributed computing is what he's referring to and in his description it doesn't have guarantees of concurrency, just guarantees of asynchrony (which is to say non-deterministic execution which may be concurrent, synchronous, or delayed in queue) – Jimmy Hoffa Jan 2 '13 at 15:21
  • 3
    I'm surprised nobody has mentioned hadoop. – Kevin Jan 2 '13 at 16:19

Are there any examples of such a system?

Yes. This pattern is known as distributed computing(or distributed programming or whatever cool word you want to put after distributed). My suggestion will be not to build this in-house before looking at other solutions. You can look at this stack overflow question for various options. And then take calculated decision.

  • 3
    Don't forget the new favorite word for this: the cloud! or cloud computing. If you do it with your own computers people call them a personal cloud or mini cloud. Marketting nonsense, this has been known as you said as distributed computing for years, just tossing out there for the OP to know where he reads about cloud stuff it's talking about exactly this. – Jimmy Hoffa Jan 2 '13 at 15:16
  • 3
    @JimmyHoffa: the word "cloud" is just as often used to indicate that your data as well as the software you use to work with it are on the internet somewhere instead of your local machine. And yet other times, it means that you're using a virtual machine that runs on a cluster rather than a single monolithic VM host; the obvious advantage being transparent zero-downtime failovers. – tdammers Jan 3 '13 at 0:11
  • 2
    @tdammers true, but none the less the type of system described by the OP is one of the many utilizing the marketing moniker of the Cloud, along with as you pointed out practically everything else. That's what makes it a buzzword, it was invented without definition so people applied it to tons of different stuff so it effectively means everything and nothing. Hooray for buzzwords. – Jimmy Hoffa Jan 3 '13 at 0:14
  • 1
    @JimmyHoffa: It's also what makes mentioning the word "cloud" here completely non-constructive. – tdammers Jan 3 '13 at 0:21
  • 3
    It could make for a good word to use against management. "What is your solution to this data processing problem?" "Well, we could distribute it to a network of computers and compute them in parallel." BLANK LOOK. "We can build a mini Cloud." "OKAY CARRY ON" – Morpork Jan 3 '13 at 1:03

As noted by other answers, this field has been known as distributed computing, grid computing, cluster computing, and high performance computing.

Let me add the distinction that, when a system can be resized after start to match the workload, it is said to be "elastic", and this is different from traditional grid computing. That is one of the (non-marketing) reasons for the term "cloud computing": the user does not need to plan for capacity, and the number and location of the machines carrying out the computation remain featureless to him as a cloud.

Also, your requirement that the master re-schedules failed tasks is called the "fault tolerance" property of that system. (Mandatory link to this cartoon)

What approach should you use to build your own, private cloud? In my order or preference:

  1. Don't build your own cloud, use the infrastructure provided by others. Amazon calls this Virtual Private Cloud, Rackspace just Private Cloud; I am sure you can find other offers and compare.

  2. Don't build your own distributed computing engine, use the engine provided by others. If you insist on using your machines, use at least as much software as possible that is provided and tested by others. You can use Hadoop from C++ via the Pipes interface or from any executable via the Streaming API. There is a similar Streaming interface on Spark.

  3. Don't code all components from scratch, use components from the community. If, for some reason, you have read so far and want to roll out your own cloud components, don't start from C++'s standard library. The main components you will need are:

    • a queuing system, as noted in a comment, to send tasks from the master to the processing nodes, and to send result confirmations from processing nodes to the master
    • a distributed file system, so that processing nodes can access the data to operate on.

    There are many alternatives for both. For queuing, RabbitMQ has a Windows installer, as does ZeroMQ. For distributed filesystems, I have really not enough experience on Windows: it looks like you can organize SMB shares into a DFS, but I can't give you any hint here. You could think, as noted in another answer, to use a distributed database such as MongoDB for the data; it does run on Windows.

You could also think about using MPI (usually the OpenMPI implementation, usually through its Boost wrapper), but notice that MPI programs are neither elastic nor fault tolerant per se; you need to take care of that yourself (at least they provide some mechanisms to achieve this). That is why I would recommend to you first to evaluate a framework for distribution that does have such properties.

protected by gnat May 30 '14 at 7:14

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.