I have an upcoming interview for a position at a company that deals with multi-petabyte scale data volumes. They're going to grill me about standard database design questions, but what are the best things to focus my revision/prep work on? How does database design change when you get to really gigantic scales?

I currently think about:

  • Indexes fitting in memory
  • In transactional tables (e.g. ad clickstream) splitting the data up into 1 table per month, or having a "recent" and "historical" set of tables with a flushing job every night or week.
  • Over-normalization - unnecessary primary keys and dimensions (such as a separate table for US zipcodes).

Any suggestions appreciated

  • I'm dealing with terabyte-scale data, so can't talk to the problems that arise at petabyte-scale. But even with the volumes that I'm dealing with, you have to think about physical limitations: just moving a terabyte across a 1Gbit network will take 3 hours, assuming you can saturate the network. Even reading into memory with a 6Gbit SATA takes a long time. I would infer that at petabyte scale, you flow the data into wherever it's going to spend the rest of its life. And of course move processing close to the data. – kdgregory Jun 4 '17 at 14:37
  • Thanks! Network bandwidth is definitely something I should think about. Want to put that as the answer and i'll accept it? – LittleBobbyTables Jun 4 '17 at 18:32

Try basic database models eg network db model and how to apply them also database relationships using primary keys

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.