If we need to store some data in a database, but without the need of advanced SQL features, can we use this scheme (written here in Javascript / node.js) :

// the DB will be in RAM !
var myDb = {};  
// read DB from disk if file exists
try { myDb = JSON.parse(fs.readFileSync(DBFILENAME)); } catch(e) { } 
 // serialize to disk every minute or when process terminates
function serialize() { fs.writeFile('./myDb.json', JSON.stringify(myDb)); }
setInterval(serialize, 60 * 1000);
process.on('SIGTERM', serialize); process.on('SIGINT', serialize);

myDb['record1'] = 'foo';
myDb['record2'] = 'bar';
// ...

See the longer version here as a gist (8 lines of code).

1) Does this DB practice have a name? Is it really so bad? Is it possible to use such a 10-lines-of-code DB system, even in production of websites that have a < 1 GB database ?

2) Scalability: until which size would this system work without performance problems?

i.e. would it work until 2GB of data on a a normal Linux server with 4GB RAM? Or would there be real performance problems?

Note: a minute seems enough to write a 2GB data to disk... Of course I admit it is 100% non-optimized, we could add diff feature between n-1th and nth writing to disk...

3) Search: can I use ready-to-use tools to do some search in such a "simple" database? Lucene, ElasticSearch, Sphinx, etc. something else?

  • 1
    What speaks against widely used nosql systems? – Knerd Dec 23 '14 at 10:39
  • @Knerd my goal was to analyse if it's possible to use a 10-lines-of-code DB system with permanancy, even in production of websites that have a < 1 GB database. – Basj Dec 23 '14 at 11:35
  • Seems like putting some sort of load on this wouldn't be that difficult. Probably 7 lines of code. – JeffO Dec 23 '14 at 12:34
  • 1
    This is only suitable for a toy system with a single user running a single thread that does anything with the DBMS. Database corruption is inevitable if the program crashes during a write. I'd be reluctant to scale this beyond (say) 1MiB, but I worked on commercial RDBMS for a quarter century, so my views are biassed. – Jonathan Leffler Dec 23 '14 at 14:36
  • 1
    Commercial-grade DBMS (not all of which are commercial — some are open source) go to great lengths to minimize the risk of damage. Ultimately, most of them can end up with corrupted on-disk data if you shoot the wrong process with kill -9 or something similar, but they also usually detect and recover because they have logging systems (write-ahead logging, etc) which allow them to spot the problem and recover the status quo ante. That protection is not trivial to implement. Remember, at a big enough scale (enough CPUs, memory, disk), things break — and break at the most inconvenient times. – Jonathan Leffler Dec 23 '14 at 23:43

Nothing is wrong with this for development. If it has a name, I suppose it would be a mock database. It is not uncommon to create a mock database that can emulate very basic functionality. You have the added advantage that you start from a scratch database each and everytime, thus you know that your program would work also for a potential empty nosql database for the same reason.

However this is not a reasonable permanent solution by any means.

Most programmers, due to the small overhead, will simply go ahead and make it work with a nosql database. It may take slightly longer, you are also programming directly to work in production and not being forced to adapt your program and test it beforehand.

Scalability is a non-issue because you're always working in development. If you crash your own computer, it is not that big of a deal. The limit of such a database would be only that of your RAM (or the RAM of the computer running the server), however I think you'd find that you'll find that the program gets very slow before you even reach the point when your program will crash.

Perhaps you could adapt some searching mechanism for the mock database, but if you're going to go through the trouble, just go ahead and use a proper nosql database. If you literally lose more than 1 hour working on this mock database, then you've wasted time.

  • Thanks a lot for your answer @Neil. I added a line of code in my question about reading from disk. (Of course, I meant a DB with permanancy, ie with reading from disk, but I forgot to write it in the question). In your opinion, what would be the main problems about using such a simple DB (see gist here: gist.github.com/josephernest/8bdf46695dc0f1f11898) in production for websites with < 1 GB database ? – Basj Dec 23 '14 at 11:33
  • @Basj The biggest problem in production without a doubt would be not being able to persist the data when you take down your application. Your client is likely going to want to, you know, see the data at a later point. If you intend on saving it after every change, you will see a lot of slowdown, and if you save every so often, you risk to lose data. Please don't use it in production! – Neil Dec 30 '14 at 9:39
  • Ok @Neil. If we use setInterval(serialize, 60 * 1000); process.on('SIGTERM', serialize); process.on('SIGINT', serialize);, do you still find it risky to use it? The data would be persistant even if we kill the application. – Basj Dec 31 '14 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.