I want to ask for a review of my big data app plan. I haven’t much experience in that field, so every single piece of advice would be appreciated.
Here is a link to a diagram of the architecture: My architecture overview
Our app is webanalytics app. I know that already there are many of them however, we use it to analyse our traffic data and we have already built-in some useful features. Our app (like each traffic analyser app) should have ability to grow and process large volumes of data. I designed a system having that principle in mind. Not everything in that plan is my original idea. I have drawn inspiration from lambda architecture (from Bigdata book by Nathan Marz, twitter architect). So after a quick introduction, let's have a look at my idea.
On the bottom, we have 'client'. This is a person which uses app to analyse their traffic. In front of the client, I put Nginx proxy to load balance traffic. I heard that node is not that good in serving static files, and Nginx is better (Apache would be equivalent, for example, Eran Hammer posted on Github article where he wrote that Wallmart uses Apache to handle https traffic, and then it proxies it to node js). Then we have node js app (running with Upstart), which is the heart of system, it has to analyse data and saves configuration to relational database (I think, that relational database is better, because in our system which currently uses MongoDB we have some ridiculous nesting, and some documents can have in one collection from 200 bytes up to unlimited space, and if we consider padding factor which mongo would create for that bigger, and then when someone edit small document it has side effect - mongo has to rewrite everything in that document and store it at the end of collection. We can set manual padding factor, however, it still feels little dirt and isn't best practice, (and probably wouldn't prevent rewriting). I have chosen Oracle, but my boss suggested using PostgreSQL because Oracle might be little costly. I thought, maybe we can go one step further and just use MySQL. That database shouldn't have any kind of big load, so let's keep it simple.
Now if you read everything, I'm very impressed with your patience ;). If you have any suggestions or you see that something is wrong, missing or want clarification, let me know, and I will respond to every feedback messages that you post.
PS. Here is an example of write and read (query) request processing. We have (on top of my plan) requests with info that page has been visited. Endpoint sends (with unique id, so when network is divided we can then, while processing remove duplicates) request to Apache Hadoop, and Cassandra. Hadoop processes all incoming requests and generates for example (format of data can be diffrent, but this is generally idea):
//by day _______ ________________ ______________ | url | views | date | | '/' | 3455 | 13-02-2016 | //by hour _______ ________________ __________________ | url | views | hour | | '/' | 355 | 13-02-2016 08:00 |
Then, when we want to get views for specyfic url, and specyfic period we don't have to query whole our master dataset, just few documents with our pre-processed data images. When someone wants to get more recent data, we send also request to Cassandra, where we want to have that kind of images too, and they should be updated with every query request (but this depends on how much data we expect to have in specyfic table/collection, sometimes it can be better to just update on every write request or don't update anything at all and don't generate images). We have also in that way 'caching' of request in 'images', and if someone changes period, we have already processed data to query on, and just have to add some little amount of most recent data.
Is connection between each part of system ok and makes sense or am I missing something important and for example I shouldn't use relational database or proxy for any reason, as I said I have only theoretical experience, and I know that someone more experienced may have some useful advices. And if everything is ok, someone who wants to build that kind of architecture can use it as example for future