I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click those product

Here is the design i am planning

  1. Whenever user clicks the product , generate the async call to produce the message on queue
  2. I am planning to use the distributed message broker and selected the Kafka for its distributed model and performance. My load is 100 to 150 request per second.
  3. Use Apache spark streaming module to parallely process the message from queue and store it in document based DB i.e. MongoDB. I selected the Mongo as i don't have relational requirement here and probably down the line i may use Sharding once data becomes big.

Let me know if my design/technolgy stacks looks good ? If not where it can be improved.

  • Voted to close as too broad because this is a fairly standard approach to processing async data, so answers won't add anything more than is already available with a Google search.
    – kdgregory
    Jun 11, 2017 at 12:26
  • However, I do have some comments: (1) also dump the data into files so that you have an audit log, and (2) give a lot of thought to sequencing and source identification (ie, make sure you handle the case where click B arrives at your processing stage before click A, because it will happen).
    – kdgregory
    Jun 11, 2017 at 12:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.