0

I'm working in a new version of an old project that I used to download and process user statuses from Twitter. The main problem of that project was its infrastructure. I used multiple instances of a java application (trackers) to download from Twitter given an specific task (basically terms to search for), connected with a central node (a web application) that had to process all tweets once per day and generate a new task for each trackers once each 15 minutes. The central node also had to monitor all trackers and enable/disable them under user petition.

enter image description here

This, as I said, was too slow because I had multiple bottlenecks, so in this new version I want to improve the infrastructure and isolate all functionalities in specific nodes. I also need a good notification system to receive notifications for any node. So, in the next diagram I show the components that I'll need in this new version:

enter image description here

As you can see, there are more nodes. Here are some notes about them:

  • Dashboard: Controls trackers statuses and send a single task to each of them (under user request). The trackers will use this task until replaced with a new one (if done, not each 15 minutes like before).
  • Search engine: I need to store all the tweets. They are firstly stored in a local database for each tracker but after that I'm thinking on using something like Elasticsearch to be able to do fast searches.
  • Tweet processor: Just and isolated component with its own database (maybe something like the search engine to have fast access to info generated by the module). In the future more could be added.
  • Application UI: A web application with a shared database with the Dashboard (mainly to store users information and preferences). Indeed, both could be merged into a single web. The main difference with the previous version of the project is that now they will be isolated and they will only show information and send requests. I will not do any heavy task in them (like process tweets as I did before).

So, having this components, my main headache is how to structure all to not have to rewrite a lot of code every time I need to access any new data. Another headache is how can I interconnect nodes. I could use sockets but that is a pain in the ass. Maybe a REST layer? And finally, if all the nodes are isolated, how could I generate notifications for each user which info is only in the database used by the Application UI?

I'm programming this using Java and Spring (at least I used them in the last version) but I have no problems with changing the language if I can take advantage of a tool/library/engine to make my life easier and have a better platform. Any comment will be appreciated.

1

You could use RPC to communicate between nodes. In my opinion that's the best way to do it. Communicate nodes directly to each other, and have the master send them tasks also via RPC.

Master node can have a table of slave nodes. Each new slave node checks-in with the master node, this is how the table is created. In the master node you can have an RPC server up and running on some fixed port like i.e. 15000. Each slave node once it's started pings the master node and reports to it by executing some method and pass in something like data from the slave node. This is how it's usually done. You can check Hadoop's IPC for more info.

  • 1
    Could you expand upon your answer some more please? – Adam Zuckerman Jun 5 '14 at 16:40
  • Please edit your answer to include the additional information. Comments are treated as second class citizens on the StackExchange network. They are not searchable (AFAIK). They are also subject to being deleted much more easily than an answer. – Adam Zuckerman Jun 5 '14 at 18:34
  • Thanks for your answer @K-Zen. I'm reading about RPC and Hadoop but I don't see how to use this in my case. I mean, in this case all nodes seem to be equals and you can send a task to any of them. In my project definition I need to know who is receiving the task. Also, I don't see how to interconnect the trackers with all the nodes. I suppose that I need to expose an API. – David Moreno García Jun 5 '14 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.