I have written applications that are very small scale and the code I write works fine for them. But I have often wondered how the server side code I write would scale up from 100s of queries per day to millions.

Also when looking at possible jobs/projects, people are often looking for developers with experience in this sort of high traffic database design so I would at least like to be able to say, I havent gotten to work on a project that was this popular, but I at least have tried to simulate it.

Are there tools or frameworks that can generate a lot of traffic or at least simulate what would happen with traffic on different orders of magnitude so I could get some practice writing optimized code for higher traffic applicaitons?

  • 1
    Your question is probably more suitable for dba.stackexchange.com most of the magic with large database comes from architecture, caching, hardware, index tuning, database design, etc. All the developer could do is utilize the db carefully by writing tuned queries and avoiding unnecessary operations. – NoChance Jul 4 '12 at 0:00

This is a common problem not only for people who have written applications that are very small scale, but also for those who did actually write large scale apps and are not able to test them. The lack of a testing environment (a very large database or data files) is often amongst developers. For example, I worked on a few different projects that were time limited:

  1. An SQL Loader which was supposed to load 15 milion rows into a table (200 columns) in less than 10 minutes. Of course until a 100 MB file from SAS platform was provided it was impossible to test the loader. Only after that I could set the right loader parameters (commits, bindsize etc..)

  2. An UPDATE on the same table with some very complicated rules (joining other 3-4 tables depending on the data that already exists) to run in 5 minutes. Of course I needed real data into the table and I couldn't fake the same row milions of times. How could I have known how fast the procedure completed without the data? After that I tunned the procedures using MERGE instead of UPDATE, deterministic functions etc..

So what I'm trying to say is that you shouldn't worry about this because you'll deal with it when the time is right. It's difficult to simulate real complex data but I'm sure that's not going to be a key skill in your employer's eyes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.