I have a node js app, in which I need to perform background jobs. My plan is to use a data structure which will contain all the jobs needs done. My design also suggests that this data structure needs to persist between application restarts. To achieve that I can think of two options to store the datastructure:

a) use a database

b) use a file

Solutions like external queues might probably be an overkill for my app(the data structure will contain 5 - 10 entries(jobs) at any time).

Currently, I am inclined towards using a log file, which I think will be faster and plus I do not need to query that info(therefore need for a database).

How do you evaluate this approach? Is using a log file for runtime persistence a good practice? Are there any other benefits/drawbacks to it? Is there an alternative I should consider in my scenario?

  • It sounds like you're confusing concepts. A log is an append-only thing, it records events. What you apparently want is some registry of scheduled jobs, that's not a log file. You can put this registry in a file but then you need to handle potential parallel accesses and modifications yourself, which is error-prone. Or you could use a really small database such as SQLite which handles all the locking stuff and is supported in many programming languages. I'd prefer the second route. – Hans-Martin Mosner Dec 11 '19 at 9:48
  • Nodejs is single threaded, so maybe I will not face parallel access issues. Unless this(file IO) is handled by the operating system which will then become troublesome. Right? Thanks, but I cannot get it out of my mind that a database is not an overkill for this as I will need to constantly update it (each time a job finishes or a new job is added). Maybe though it is the only reasonable solution !? – Cap Baracudas Dec 11 '19 at 9:56
  • Sqlite persists after system reboot right ? But it is polled on the memory for faster access ? – Cap Baracudas Dec 11 '19 at 10:06
  • 2
    Back in the day, batch jobs were done with single files in a directory. All jobs were done, when the files were moved from one directory to another. In case of errors it was moved to another directory. I would call that poor-man's-queue but it works. So a file approach could be viable under circumstances. – Thomas Junk Dec 11 '19 at 11:36
  • @Hans-MartinMosner under certain circumstances a log-file could do the trick: If you know beforehand which jobs have to be done, reading the logfile could give you a clue of what is left to do, But that is - admittedly - not the best of all solutions. – Thomas Junk Dec 11 '19 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.