2

I have done quite an amount of research on the topic so far, but i couldn't come up with a conclusion to make up my mind.

I am designing a social network and during my research i stumbled upon graph databases, i found neo4j pretty interesting for user relations and traversing through nodes. I also thought of using a relational database such as MS-SQL or MySQL to store entity data only and depending on neo4j for connections between entities. Of course this means more work in my application to store and pull data in and out of 2 different sources.

My first question : Is using this approach (graph + relational) a good approach for designing my social network keeping in mind that users on social networks don't have to in synch with real data by split second ? What are the positives and negatives of this approach ?

My Second question : I've been doing some reading on CQRS and as i understood it is mostly useful for collaborative environments, and environments where users see a lot of "stale" data. social networks has shared comments, events, etc .. and many users query or update the same data. Could CQRS be a helpful approach ? Would it give any performance/scalability benefits or non-useful complexity ? Is it fairly applicable with my possible choice of (graph + relational) databases approach mentioned in the question above ?

My purpose is to know if the approaches i have mentioned above seem good enough for the business context.

  • "Your questions should be reasonably scoped. If you can imagine an entire book that answers your question, you’re asking too much." (help center) – gnat May 27 '14 at 13:06
  • Fine :) I'll trim it down to the first two only then, they circulate around the same topic, i am only looking for answers evaluating my mentioned approaches ! – Siraj Mansour May 27 '14 at 13:12
  • @SirajMansour Have you read Implementing Domain-Driven Design? It is a valuable resource for doing DDD and CQRS. – Songo Jun 4 '14 at 13:08
  • @Songo i will look into it then ! Thanks for the suggestion :) – Siraj Mansour Jun 4 '14 at 13:10
3

I'll weigh-in with some brief thoughts.

Topic 1 : Graph databases are good at modelling/querying hierarchies. Say that in your social app you want to let users know whether any of their Friends - or any of their Friend's Friends - have a birthday today. That can be an enormous query if you're recursing down all levels of Friends. A graph database should do this better than a relational database.

Yet a relational database is really good at a lot of other things - so you might consider utilizing both - relational for your general purposes, and graph for special purposes.

Topic 2 : CQRS is an architecture that helps with highly concurrent systems. In a brief nutshell, writes are thought of as a different problem than reads. Writes are typically dropped into a queue (fire & forget) and picked up when the system is able to / in a load-balanced manner. If an error such as a deadlock is encountered, the write request remains on the queue and is retried until it (hopefully) succeeds (this is "eventual consistency").

  • To me CQRS seems convenient for a Social Network, do you think so ? – Siraj Mansour May 27 '14 at 15:16
  • 1
    I think it depends on envisioned scale. If the app is open to the public and you want its popularity to explode, then CQRS may certainly be worth the investment. On the other hand if the app is meant for something smaller like a single org/sports team/Cub Scout troop, then concurrency won't be a huge problem - and you can probably skip CQRS. – Dave Clausen May 27 '14 at 15:26
  • Fair enough :) ! – Siraj Mansour May 27 '14 at 15:26
  • Your overall question is timely for me :) as our SaaS app is now reaching the scale that we believe we need CQRS. We're looking at a product called NServiceBus to take care of the CQRS queuing/retry/threading/prioritization plumbing. Thankfully our app's write operations are pretty well-designed using a Repository pattern, and because of this (and NServiceBus), we think/hope it will be a relatively painless transition to CQRS. – Dave Clausen May 27 '14 at 15:39
  • it is said a lot around that i should deal with scalability issues the time they show up, but here i intend to take a different route, because even if the product am designing fails to succeed as an idea for the end-user i would love to have gone through this technical experience. – Siraj Mansour May 28 '14 at 10:56
3

In my opinion, you are over-engineering the project. I think you do this because you believe you have to rely on cutting-edge techniques to handle business scale, but in many cases you will be better off relying on proven techniques and innovating only in a very focused scope.

A word of caution on graph databases: in my experience, they promise more than they can deliver. My experience is now some years ago, so I can't tell you if they have matured as products; but do you want to find out if something scales by using it as your main workhorse?

Let me remark there are some alternatives for such graph algorithms, some of them with provable scalability because they build on Hadoop's HDFS: see this SO thread or this Spark library.

On the topic of CQRS, it seems to deal with the kind of problems that large websites traditionally handle with a cache layer on top of their database replica sets. Write a wrapper around your queries that first looks into the cache layer, and if that misses its mark, then pulls the data from the database and also writes the result set into the cache. Here is a simple example in Python.

Moreover, splitting your queries into Commands and Queries on top of two database engines means that you have to decide, for each user request, whether it is graph-related or not, and whether it writes or just reads something; usually you will have a mix of all four possibilities. If you get your decisions right, you will get a faster, more responsive social network; but be aware that you will get the same performance boost by making the right decisions in almost any language and runtime. And even so, response times will almost certainly be dominated by network latency.

In your place, I would concentrate on one of both topics, and I would also concentrate much more on the question: what does this technique enable that is better than existing social networking sites?

  • but as far as i know Hadoop itself isn't designed for real-time querying but for background analysis. As for graph databases, the guys at fiftythree.com seems to have gone through a pleasent experience with it using neo4j, they hit 10,000 requests/sec. making.fiftythree.com/load-testing-an-unexpected-journey – Siraj Mansour May 27 '14 at 15:24
  • HDFS is decoupled from Hadoop. Spark has a lot more emphasis on stream processing and is therefore faster (if you choose your caching strategy right, of course). I can't say for Hama. Storm is another engine on top of HDFS that could be more geared to your needs than Hadoop. – logc May 27 '14 at 15:28
  • With regard to graph databases, this depends on the size of the graph that you expect to handle, of course. I can tell you that millions of nodes and hundreds of millions of edges can be ... troublesome, with some of them. Please regard this as what it is: a word of caution. – logc May 27 '14 at 15:40
  • I was wondering if you have any benchmarks or any one with a current experience running ([HDFS + Spark] or other suggestions) smoothly for real-time querying ? That would help a lot :) – Siraj Mansour May 28 '14 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.