suppose im writing a dating website, similar to okcupid. there are profiles, and i need to compute the (N^2) "match" table - given every 2 profiles whats the match between them?

I was thinking this could be done by creating a spout to listen on a "new/updated profiles" queue (say kafka, doesnt matter), but then how do I break down the matching to achieve any degree of parallelism?

If I have a single bolt that compares the profile vs the entire DB that wont scale.

If I create another spout, for "all profiles" it will run in a continous loop and never stop (?)

Obviously the assumption is that the "churn rate" (rate of new/updated profiles) is less of an issue than the sheer size of the database.

Any suggestions on how to design the topology would be very welcome.

  • How do you define "match"? – Vladislav Rastrusny Mar 16 '15 at 15:45
  • @FractalizeR - do some sort of calculation that results in a match % - 0 to 100 double value of something. this match value will be stored in some sort of key value store where the key is a combination of 2 profile IDs or something – radai Mar 16 '15 at 16:20
  • There is no possibility of answering your question without knowing what exactly is your matching algorithm. May be it is so simple you can run it as a simple SQL on your database. It could also be so complex, you will be required to calculate it on each unique tuple of profiles. – Vladislav Rastrusny Mar 17 '15 at 21:09
  • why? just assume its complicated enough to be impossible using a sql query. it needs to read both entries, and do something to arrive at a result. im asking about the infrastructure setup for storm. – radai Mar 19 '15 at 8:08

The obvious answer to this problem is "don't do N^2", at least not for the full value of N - you can very rapidly reject large parts of your data set for all sorts of reasons:

  • Wrong sex. While I'm a liberal kind of guy, I'm still not interested in dating other guys. There may obviously be some exceptions to this - e.g. bisexuals.
  • Wrong country. It doesn't matter how hot they are, I'm not going to want to date someone in Australia if I'm based in the UK. Again, there may be exceptions but most people aren't looking for a long-distance relationship. (More generally, get people to say how far they're prepared to travel for a relationship. New York to California probably isn't going to work for most people).

There's probably lots more criteria I haven't thought of yet - by doing this you've hopefuly solved the "sheer size of the database" problem and got it down to something tractable.

As an aside, I suspect you'll find Data Science of Love interesting viewing. Disclaimer: I know the presenter personally, but have no professional relationship with either him or eHarmony.

  • yes, i know i can reduce the problem in various ways. the question though was about how to design it so that matching a single profile can be done in parallel. – radai Mar 17 '15 at 18:53

Although I have not worked in this area, it seems like this is the type of problem one would use something like Apache Giraph to handle. You essentially want a huge population of "things" which you suspect are "related" to each other possibly across a large number of dimensions, and you want to pair them up optimally based on the strongest matches.

You don't want to hard code what those things are, the possible dimensions they may be related, and the rules that define each relationship. Not only would that be a lot of work, it wouldn't scale well.

This link is a proprietary graph product, but does a good job showing how a graph data model would apply to online dating. Same concept could be done with Apache Giraph, or any other graph computational tool:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.