1

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only processed once by a single server.

I've got X servers, each with a unique id (int), each polling the database for the next Y partitions (pre-read and buffer, then loop and process the pre-read partitions, continue until no more partitions remain).

I can see in the log that I get many clashes, eg multiple servers trying to process the same partition and failing, when trying to take ownership (as it has already been taken by another server)

All these fails are waste of time, network round trips and compute power.

I'm looking for ideas on how to split the partitions among the servers when reading the partitions.

Each partition has the following attributes:

  • Id - string[13]
  • Sequence - long (incrementing counter)
  • Create time - timestamp

Any ideas on how to best implement a non-clashing read algorithm ?

Keep in mind:

  • Number of partitions are unknown
  • Number of servers are known, but may increase/decrease
  • I can modify/add attributes to the partition if they can help minimize clashes
  • X Partition should not have affinity to Y Server, any server should be able to process any partition

My Idea: I've been playing around with the idea of using the server id to offset their read, eg server 1 reads 0-1000 records, server 2 reads 1001-2000 records and so forth, however too many issues occur, there might not be partitions enough to divide on X servers, or the servers may be started at different times reading the same partitions even with an offset according to their server id.

Improve this question
0

2 Answers 2

Reset to default
0

Use a hash mod number of servers. Check if it equals your server id. Only look at other partitions when there is no work for yours. You will get some collisions but only when you've run out of work.

Improve this answer
4
  • Would it be possible to somehow map between the server id and partition sequence. Eg server id may be 3 and partition sequence will be 1,2,3,4,and so on. If somewhow we can map between the two I can avoid adding a new attribute for server id on the partition.
    – Kelvin Wayne
    Jul 24, 2018 at 10:28
  • My suggestion doesn't involve adding any extra attributes to your data. Avoiding collisions is not a matter of correctness, but performance. Your existing algorithm is pessimistic, because each server is looking at the same input Q and taking data in the same order. Anything to randomize the selection choices by the servers will alleviate that problem.
    – Lewis Pringle
    Jul 24, 2018 at 14:23
  • If you have N servers (lets say N=3 for an example), and lets say your input Q is in a SQL database and you select the work with select top 1 from worktable; just change that select statement to select top 1 from worktable where workitem.id %3 = 0 (for server 0), select top 1 from worktable where workitem.id %3 = 1 (for server 1), and select top 1 from worktable where workitem.id %3 = 2 (for server 2). Each server will get different items at essentially no extra cost. You can even have them fallback to the original query when no work is available for their 'mod' check.
    – Lewis Pringle
    Jul 24, 2018 at 14:25
  • got ya, great idea.
    – Kelvin Wayne
    Jul 24, 2018 at 20:30
0

I've achieved this sort of distributed workload by using JMS like so:

  1. Primary program reads source data and composes individual "messages" of work and posts them to a JMS queue
  2. Worker servers read messages from the queue (this can be wrapped in transaction handling if desired) one by one until the queue is empty.

No need to partition the work, since any work unit can be handled by any server... just compose the work units at the desired granularity and let the JMS server handle distribution to your worker servers.

Note: The "J" in JMS stands for Java, but really there are JMS clients for many languages.

Improve this answer
4
  • We did look into a queue system, but abandoned the idea and instead chose to use a database. One of the reason being, the only queue system in our enterprise mature enough for our purpose is one, we do not wish to create a dependency to - due to license costs.
    – Kelvin Wayne
    Jul 24, 2018 at 10:06
  • @KelvinWayne, you're not able to just run an ActiveMQ server (it's open source) or something? It really sucks when the right technology exists, but you can't use it due to arbitrary constraints. Maybe look into ways you can implement a queue on a database using careful locking?
    – Maybe_Factor
    Jul 25, 2018 at 0:48
  • I could, however the scale we will be running it at would require a complex installation that needs to be maintained, regardless of whether we chose RabbitMQ, ActiveMQ or some other free technologoy. We also looked at cloud paas solutions, however they do not (yet) provide selector filters.
    – Kelvin Wayne
    Jul 25, 2018 at 10:26
  • If we were to use a queue system, we also have WebSphere MQ on Z/OS but due to multiple reasons, we have decided to use a DB. We already have multiple DBAs and a Always On Availability Group MSSQL cluster setup, so even though you are right in terms of technology match, using a DB is a better match for our corporation.
    – Kelvin Wayne
    Jul 25, 2018 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.