How do I evaluate if a data warehouse is necessary?

Question

I need to put data from MongoDB together with tracking data from a quiz app (stuff like which users clicked what button etc). Then I’ll use this in machine learning.

I looked at all the DW options such as BigQuery, and I’m asking myself if a SQL database wouldn’t be enough. Is there a general strategy to evaluate when a DW a must-have, or what happens in contrast if I just use an SQL data base?

Answer 1

Some patterns only make sense at a scale. It often makes sense to separate analytical from transaction workloads, but not everything has to be cloud-scale. Many gigabyte-scale problems can be efficiently approached e.g. by loading data into a local database (such as Postgres or SQLite) and performing queries there. In contrast, cloud based approaches make a lot of sense when:

• the data is too much for a single computer
• your data is too much to efficiently transfer, so you'll have to take the compute to the data (instead of bringing the data to the compute)
• the data cannot be structured for efficient queries, so reasonable query performance relies on massive parallelization (e.g. with a Map-Reduce architecture, or search products like Amazon Athena).

Don't get tricked into paying for an expensive cloud product that you simply don't need.