I want to create a aggregation job that executes a big db query and flush it into BigQuery. My question is should I include only the id of the entities (campaign id, advertiser id, user id) or should I include also the meta data (campaign name, advertiser name, user name).

The upside for metadata storing is that I have the whole data in one place. The downside is that when the metadata been changed (campaign name modified for example) then the whole data should be aggregated again.

Is there a best practice for this kind of situation?

  • Could you have regularly scheduled updates to BigQuery? Jan 19, 2021 at 14:20
  • yes, everyday new data will be inserted
    – Avi L
    Jan 19, 2021 at 15:22
  • You may also need an update process to handle metadata that has changed. Jan 19, 2021 at 15:35


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.