I need to architect a system for my company. My question is specific to logging. Part of the requirement is to log 2 kinds of data.

A1. User events (Ex: UserA clicked on Button1, performed Action#21. Error occurred while performing Action#34, Successfully completed Action#51).

A2. Complete API traffic(request and response payload) logging.

A3. Develop a dashboard to display analytics of user events (from A1 above).

We are mostly considering open source solutions without licensing requirements (except Oracle). Our current plan is this:

B1. Use an Oracle 11g DB as application's primary data store (User & access management, shopping cart, order management etc.).

B2. Use a MongoDB or other NoSQL DB to log user events (from A1) and log complete API request and response bodies (from A2 above). Reason is, we believe RDBMS(Oracle) will be slow for writing and reading huge payloads in BLOB fields. NoSQL might be better for these write heavy operations.

B3. Use a logging microservice, so that our server app can POST the payload to the microservice and forget about it. The write heavy operations will be handled by this microservice so that our server app does not take the added overhead of logging and remain highly responsive.

B4. Develop our own dashboard to show customized graphs and stats on user events (collected using (A1) step above). But the events table could be huge (millions of records) to query from. So not sure if its the right approach.

Following are my questions:

  1. Would you all recommend sticking with the above plan or if you suggest doing something different, what would it be?
  2. What I lack knowledge in is "BigData". What exactly is BigData in a nutshell? Can we apply it for our needs? Are there open source solutions for BigData?
  3. Would using BigData or any other solution help with faster analysis/querying of data for our analytics dashboard?
  4. Is there an out of the box open source solution for dashboards(UI) so that we don't have to build it from scratch?
  5. If sticking with NoSQL is a good plan, what NoSQL DB would you recommend? MongoDB? Cassandra?

P.S: We will be purging data older than 6 months from the logging tables.

Estimation: Users: 50k max. Transactions/Requests: 500k per day max. Size of payloads: 25KB per req/res avg.

Please feel free to criticize the entire plan and suggest an entirely new one that you think would be a better solution(using open source tech).

Here is a high level design of our plan:

enter image description here

  • 1
    What is completely missing from your plan is an estimate of the request rate and data throughput so that you can derive proper requirements which may inform your technology choice. Database hardware can have a massive effect on performance so that's worth discussing as well. I'd also question why you have to persist all requests+responses. That's highly unusual, raises interesting privacy questions, and amplifies technology problems. In most scenarios, only logging business events or sampling a few sessions to be logged might have similar value at a fraction of the cost. – amon Jun 23 '18 at 20:13
  • @amon I just added traffic estimates. Persisting req/res payloads is only on dev env and not on production. Every user will go through a 100 API calls to complete a transaction(several of them are called in parallel) and we need a way to track issues and fix them on dev environment. – user1220169 Jun 25 '18 at 12:45
  • The numbers in your question suggest a total data set size of 2TB and a peak write rate of <2MB per second. That looks very manageable with any setup. Or does that have to be multiplied ×100 for the API calls per transaction? But if response logging is only done in dev, is scalability and performance even important? I think I'm a bit confused because A1 (analytics) and A2 (dev response logging) seems unrelated? – amon Jun 25 '18 at 13:08
  • @amon sorry i must've been clear. We'll have both dev as well as production environments. The only difference between them will be "no API req/res logging in production". Others would fairly be the same. Your assumption of <2MB per second is more or less right - it could be 5MB/sec at max in prod. – user1220169 Jun 28 '18 at 20:30

This might be a partial answer but I think there's some info that might be useful in it, so...

First, I'll try to answer the questions:

  1. "Would you all recommend sticking with the above plan or if you suggest doing something different, what would it be?"

I would actually try to convince the stakeholders (money payers) that using an enterprise software would be in their best interest 'cause you're describing a very hard problem and you'll likely spend more money developing it than something like New Relic costs.

That said, the implementation totally changes based on what you're currently doing for releases and if you control the application code you're trying to monitor or if it's some 3rd party plugin you want to get data about too. I'm going to guess that the app is your company's and that your might have some CICD but it's not 100%.

If that's close enough, I think it would be easy enough to stand up a Kafka cluster or three and log everything you can get your hands on into them. Maybe one is for RESTFUL calls and one is for app-level errors or whatever you decide. This could stand as your centralized logging system.

Since you control the application code, you can log pretty much anything you like by asking the developers to add more logging, where you want. That's not usually a favorite request but if it makes sense and especially if higher ups back the idea, you can probably get all the logging you want to get. That will tell you about RESTful calls, app errors and things like that but you'll have to go deeper to get latency reporting and info on slow SQL queries and all that. I believe you can get some good info out of some free Nagios logging too. At a minimum, you should be able to get stats about how the hosts are doing; e.g. Memory available, CPU consumed, Disk space, etc.

You can then have something consuming all these messages and tie in some pretty useful automations, based on the state of your system and the actions coming in. "Oh look, everyone is buying all the X-product! Let's make a promotion to capitolize on it!" or "Oh look, all of our apps are throwing 500s! Let's go hide somewhere until this all blows over!"

I'd say you should use something like AWS's SQS because you don't have to worry about managing the Kafka/Zookeeper cluster on top of managing your application(s) code. SQS is pennies for millions of messages but if you're going 100% open source, Kafka is a solid choice, in my humble opinion.

Choosing NoSQL...that one's tough here. I think it's probably the right choice but maybe, if you can, hold off on deciding about that level in your system until you find out how you want to use that data, in other words, make sure the use case fits NoSQL before you commit. Switching and keeping any history would be tricky.

  1. "What I lack knowledge in is "BigData". What exactly is BigData in a nutshell? Can we apply it for our needs? Are there open source solutions for BigData?"

I'm no PHD but I dabble and I work with those teams, daily. With that history, I'll try to explain in my words. Big Data is a blanket term you can use to talk about problems and solutions with, when the problem you're trying to solve has a grip of data tied to it by nature. It's usually not helpful to work from this mentality when your data is still in the GB range but I'm sure there's plenty of use cases where it makes sense.

Anyways, it works like this; Say there's this thing that we all do, like use the internet. Pretend, if you will, that there was some way to collect data about our internet usage, like what sites we visit and what search term we used to get to that site. That's a looooot of data and when you think of it and use it as kind of a resource, you get a lot of insights and benefits out of it but it's also really hard to sift through all that data and reason about it. In this example, the data is all our internet usage. The problem is how to sale crap to us, using google adwords or whatever, and the "Big Data" in this is that we're considering all this data and how we can use it. A big part of "Big Data" is also the infrastructure used to run it. It's not trivial so it usually gets a lot of attention.

There's lots of open source tools you can use for using your data. If your apps were to use any tools, Spark comes to mind. You could set up streaming from your logging message system to modify all the incoming data so that it's easier to use for humans and machines. I guess if you get really ambitious, you could try to implement some machine learning based on all your data. You'll have to come up with the problems you want to solve first, though.

  1. "Would using BigData or any other solution help with faster analysis/querying of data for our analytics dashboard?"

Yes. That's the whole point of it all. The problem is, you have to hire a bunch of expensive people to implement it. Your best bet is to tackle that hurdle when you get to it. If you can create your logging system and the dashboard, you'll be worlds better than you are now. No need to build an F1 car when you just need a tractor, ya know? And if I were to continue the F1 car and tractor metaphor, you can think of it like this -> The tractor that you're going to build now is creating the paved surface for your BigData F1 car to do impressive things with. You'll need all that data your logging system is going to collect before you can use the data to reason about, if that makes sense.

  1. "Is there an out of the box open source solution for dashboards(UI) so that we don't have to build it from scratch?"

I don't know of one but that doesn't mean it doesn't exist. Building the dashboard can be as big or small a project as you make it. Honestly though, I think this is the easiest part of the work ahead of you. Enjoy, if you build it yourself! It should be fun.

  1. "If sticking with NoSQL is a good plan, what NoSQL DB would you recommend? MongoDB? Cassandra?"

My opinion here is just to use a cloud tool and let them decide the engine. I doubt, from what I'm hearing, that you'll be taxing any graph database on tough queries. Your issue is most likely going to be from reads and writes, so maybe figure out how much you think you'll be pushing into the DB and how you'll be querying that data. It should lead you to whether or not NoSQL is a better fit and during your reading, I bet you'll stumble on some article that makes up your mind on which engine you chose.

It's an interesting problem and it sounds really fun to work on.

Good luck and I hope that helps, at least a little!

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.