-3

I need to put data from MongoDB together with tracking data from a quiz app (stuff like which users clicked what button etc). Then I’ll use this in machine learning.

I looked at all the DW options such as BigQuery, and I’m asking myself if a SQL database wouldn’t be enough. Is there a general strategy to evaluate when a DW a must-have, or what happens in contrast if I just use an SQL data base?

2

Some patterns only make sense at a scale. It often makes sense to separate analytical from transaction workloads, but not everything has to be cloud-scale. Many gigabyte-scale problems can be efficiently approached e.g. by loading data into a local database (such as Postgres or SQLite) and performing queries there. In contrast, cloud based approaches make a lot of sense when:

  • the data is too much for a single computer
  • your data is too much to efficiently transfer, so you'll have to take the compute to the data (instead of bringing the data to the compute)
  • the data cannot be structured for efficient queries, so reasonable query performance relies on massive parallelization (e.g. with a Map-Reduce architecture, or search products like Amazon Athena).

Don't get tricked into paying for an expensive cloud product that you simply don't need.

2
  • Thank you! Could I only do a “staging” database for my analytics? This type of database is created prior to running my analytics and then flushed of data when the analytics work is done for the day. The next day (or week or month) it is re-created with all of the available data.
    – a0142204
    Apr 29 at 16:16
  • @a0142204 That can be sensible IF you have sufficiently little data so that importing the new data can be done quickly and cheaply (considering data transfer speed, traffic costs, and time spent building indexes in your DB).
    – amon
    Apr 29 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.