4

I have to develop a logging system for my project which has multiple features. I am stuck at a choice. I could either make a centralized database and front end for everything. Or I could have one log table for each feature and a common front-end none the less.

Centralized seems like the way to go but it has its disadvantages.

  • Some features create more log entries while others don't. So the ultimate table can have a million log entries for feature x and 23 for feature y. So every feature pays cost of access time for greedy use of one feature.

  • Also one table can grow pretty big.

  • And I will have to standardize one single log format and I cannot mold it for individual feature while still making use of DBMS's capabilities like sort, join, group etc. I'll have to store all the feature details in a json data field.

Downsides of a distributed system include

  • I'll have to maintain multiple tables.

  • There will be no standardized schema so building a common front-end will be that much troublesome.

Does anyone have any experience with this sort of thing?

4

For the most flexibility, do a combination of both: a single centralized table and individual tables that handle data specific to a particular action or process.

Have one central table to log all basic actions. This could be an individual entry 'Account Payable - Add' or a batch 'Payroll Process.' There could be separate table(s) handling more details.

This way, you can find what a user(s) did during a particular timeframe without too much trouble. As this table grows, there's no reason it can't be managed by purging or archiving data.

Having a table for a particular process or even for each table in your database to track changes allows you to capture the data in a way that makes sense, but has some sort of CentralActivityID for linking.

This will be much easier to have your logging match what the application is doing now and in the future. Not every process will require its own detail table either.

0

I am no expert in logs or database. I would use a logging system like logstash for this, but to answare your question, using a single table with indexes on functionality and level is the best choise, since indexes will give you speed, while having one table will give you flexibility. For example you could query all errors, or all log entries from one user, so you could debug the system in a cross-function way when needed.

  • Yeah that's a very good point I mussed. A single user's system wide logs would be a nightmare in distributed design. – Achshar Dec 18 '16 at 23:09
0

It's a bit hard to answer the question without knowing what you'll be using the logs for.

If logging is purely for internal debugging and troubleshooting, then per-feature logs might make sense, as you'll be zooming in on particular sub-system anyway.

If logging is used for customer-facing reports or dashboards in your product, then you'll need to consider foreign key relationships between actions in different features (e.g. the user performed actions X and Y during the last 10 minutes that resulted in Z), which in turn pushes towards a single table (e.g. with NoSQL DB), and/or normalized schema.

  • The intention is for them to be user facing but there are no cascading effects like you describe, the logs won't have any foreign key based information, they will be self contained and only used for manual correspondence. My users end up lost / play the blame game. So the idea is to establish clear blame. I don't have any intention of creating reports and such. Although having the option to do that in future if needed would be a good thing. – Achshar Dec 19 '16 at 0:16
  • So it's troubleshooting. In that case, quick'n'dirty does the job. I'd go for whichever way is easiest to build, which in your case looks to be component-specific logs. – RomanK Dec 19 '16 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.