0

Right now we have a highly analytical system which collects tons of various metrics. We need to display this data in a web ui for users to consume.

Since we are in early stages of development we are throwing all data into google bigquery, when a users visits the app we run a cloud function to request the data from bigquery, and store/cache it in firebase.

When a user visits the same route for the same view we will show them the data from firebase (old data) whilst we fetch any updated results from bigquery.

The data itself is similar to web event logs for the most part, but some of it is more complex, and we are storing it as a json string in bigquery.

It feels like a complex and hard to maintain setup for a small team, as we are still building the MVP and keep iterating on the views presented to the user. My question is how to simplify our database choices, to have to maintain less DBs, but still have flexibility around changing the final views and an acceptable level of latency for queries <2secs.

2

Welcome, not quite the questions I think will be accepted here, but never the less; my 2 cents follows.

Maybe look at influxdb, I have some success working with them.


For prototyping, I don't think you should work with anything more then what you already do, as you don't need to optimize queries at this stage, you have a ux/ui to build.

If your project is successful you will work daily with db optimizations in the future, or so I guess at least. At a later stage you therefor probably want better suited technologies to rely on that you can test and configure between. An ORM or a smaller repository layer is therefor worth thinking about. You don't want a final decision on your persistence engine, you want a flexible solution so you can change between solutions as technology evolves. There is no one correct answer to that question.

Before you start looking for better technology for persistence, build a cache layer to decrease load. Less solutions to research, less difference in choice. I like redis, but what ever...

I suggest you look at event-sourcing with eventual consistency, and use a cache layer to keep track of the current state of the data. I promote this methodology as I notice that the time dimension of data is more and more relevant in everything I do, and I don't know it until I learn it. As you are writing an analytical system, I believe you would agree.

  • Thank you for your answer and apologies if this is not the right place for this question... one follow up would be - influx looks good but what about anything that is not timeseries? Would you throw it in there also? – dendog Jul 31 '19 at 16:28
  • @dendog Everything is related to time in an event-source system. In an event-source system you can only read and write. You write events, that are always identified by time. You read from the event-source system to (re-)build your current state in the cache layer that can be both lazy or eager loaded. So the answer to your question is both "yes" and "no", I would use a pattern that allows me to build my states differently how ever I like in the future, which is where you work with the data. We can have a chat if you like to know more, it's a bit hard to discuss like this in the comment field – superhero Aug 1 '19 at 9:29
  • Sure, that would be great - how? – dendog Aug 1 '19 at 14:42
  • You can join this chat and we can talk when we are both online chat.stackoverflow.com/rooms/197362/eventsource-talk – superhero Aug 1 '19 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.