7

What would a system like BOINC look like if it were written today? At the time BOINC was written, databases were the primary choice for maintaining a shared state and concurrency among nodes.

Since then, many approaches have been developed for tasking with optimistic concurrency (OT, synchronization primitives like vector clocks, virtual synchrony, shared iterators etc.)

Is there a paradigm for optimistically distributing units of work on sparsely distributing systems which communicate through message passing?

Sorry if this is a bit vague.

P.S. The concept of Tuple-spaces is great, but locking is inherent to its definition.

Edit2:

The entire system is sparsely distributed - they can communicate only through WAN. And communication can be slow and faulty. The question is about how to best distribute units of work among them without a central co-ordinator and with as little consensus as possible (because consensus is expensive).

The answers here seem to be talking about databases - data isn't the problem. The problem is in distributing work.

Edit:

I already have a federation system which works well. I'm looking to extend it to get clients to do units of work.

3 Answers 3

Reset to default
2

When you know what your system really needs is when you can choose the better solution. The biggest data systems are made for great amounts of data, if your system has little data (GBs) then one typical solution will fit. You must think that big sparse data systems require special data models that are very different from SQL typical data models.

RDBMS data models look forward to represent a consistent state in every moment, distributed systems don't (see CAP theorem).

1
  • Thank you for your answer. Data is not the problem here - the system has eventual consistency, and leans more toward availability. The problem is how to use whatever data model exists and use that to assign work to distributed units.
    – Asti
    Commented Oct 31, 2012 at 6:42
0

The easiest solution is usually the best. There's nothing wrong with a central server distributing work to a number of workers provided you can make the jobs big enough that the overhead is not overwhelming. There's no point redesigning BOINC just because fashions have changed.

9
  • 1
    The easiest solution is easy precisely because it puts a set of constraints on the working of the system, reducing the system complexity. E.g., Having event handling code running on the GUI event-loop is the easiest solution. When the code is computationally intensive, you cause the GUI to become unresponsive. You will have traded the easy solution for the constraint that you will have an unresponsive GUI.
    – Asti
    Commented Jan 15, 2012 at 14:54
  • I appreciate the honest answer, but it is akin asking for a solution like Operational Transformations for collaborative editing and being told that you should stick to database transactions because it is easy.
    – Asti
    Commented Jan 15, 2012 at 14:57
  • @Asti, except that database transactions aren't a valid answer when you are looking for Operational Transformations; it doesn't solve the problem. What do you lose in this case from having a centralized architecture? Not enough to warrant the increased complexity.
    – dan_waterworth
    Commented Jan 15, 2012 at 15:14
  • All the conventional problems of a Master-Slave architecture, Dan. Single point of failure, bottle-necks due due to load on master, messages required to acquire a lock, reentrancy, task deadlines... I'm looking for a shift which could mitigate classic problems early on.
    – Asti
    Commented Jan 15, 2012 at 18:16
  • But you inherit all of the problems of a master-master architecture, complicated error cases etc. There's no need for locking, optimistic concurrency is possible.
    – dan_waterworth
    Commented Jan 15, 2012 at 19:09
0

You're asking if programming is like clothing industry so if you won't redesign the system each 2 weeks your friends will laugh at you because you lost sync with fashion? :)

If it was written today probably marketing team would stick a big "cloud" banner to it, it'd require 20 amazon nodes to run at decent speed and would break at least 5 times daily because of using latest-and-greatest "nosql storage". Then you'd face scalability problems because those cloud systems usually don't scale out of the box as they promise, and you'd be forced to re-think the design.

Today it's for sure a little different. You could for example manage the state using some mature nosql solution like memcached for staging and then just put the data into SQL. What's wrong with the old approach if you can get a server with 64GB Ram and 24 cores for something like $300-$500.

1
  • My friend, I'm looking for a paradigm like Tuple spaces, not a set of programs that I need chain or a database to run on Amazon or Azure.
    – Asti
    Commented Jan 16, 2012 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.