Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
A company I'm going to interview with says they deal with huge amounts of data. How does one prepare for an interview where that fact is telegraphed beforehand as something that one really ought to ponder over?
Should I memorize maximum file sizes for Ext4 file systems or something? Or just be aware of the need for the best possible algorithms...
Thanks.
I worked at a storage company and it sucked immensely. Basically data is stored on large hard drives which are attached to servers. In addition they attempted to have a clever scheme for data integrity/security/recovery etc. Hard drives are dirt-cheap, but they are very likely to die within 5 years. So, how do you keep the data safe?
Google came up with this back in 2003:
http://en.wikipedia.org/wiki/Google_File_System http://labs.google.com/papers/gfs.html
The company that I worked for never got even close. As the result hospitals lost important documents, CIA lost track of terrorists, banks lost customer data - I exaggerate not!
I highly recommend reading the Google File System paper. They did a good job at explaining stuff.
Also, be able to compare different file systems (at a high level): ext3, ext4, reiserFS, NTFS, the one that Apple uses ...
Also try to learn what a company Akamai does and why they are in business - who would pay them and why.
Knowing something about networking, caching, proxy servers, BGP tables http://en.wikipedia.org/wiki/Border_Gateway_Protocol would help.
Look up what the company does, try to figure out who their customers are, what problem they have and why they pay $$$ to your employer. Then try to imagine what low-level details are involved in streaming stuff from point A to point B, including protocols, etc.
Hopefully this will help you 'WOW' them.
EDIT: Do not be cocky at the interview, even if you happen to know some things that they do not, do not show off. You are trying to get a job, right?
I realize that my answer is not complete, albeit somewhat popular.
I would add knowing the difference between TC and UDP, and when you would need each one.
I would take this with a pinch of salt unless you get some concrete figures such as number of Terabytes and number of locations.
In may experience most sites think they have a "huge" amount of data because they are inward looking and have no idea what the rest of the world is doing. Currently I am looking into a system with a billion rows in the database, and I would classify this as a "medium" to "large" system.I have worked on systems where the transaction history alone held millions of rows.
Never trust terms like "huge", "large", "high volume" etc. always ask for concrete numbers. In most cases these will turn out to be rather modest.
We at our company deal with a lot of data (social stuffs), what I have learned is:
I know this is not the answer you have expected but I hope it would be of some help and good luck.
It depends on the job.
If you're doing database work, you should know your joins so you don't create a massive Cartesian Join / bog down the system. This is a big problem with some of the developers at my workplace.
Knowing how to string together subqueries and writing efficient code is always a plus. Our team prefers programmers who don't need to test every single snippet on the db2 server. Being confident in your code prior to testing keeps you in good graces on my team.
Safe bets would be network topologies and technologies, as well as application architectures. High-volume filtering and routing techniques might also be good. If their primary business involves serving or storing data from external customers, then security should be involved. Showing some database knowledge is probably a good idea too. If their data is transactional, then you should know basic transaction semantics.
