0

I've got a program for analyzing specific folders / websites. The program always does the same steps for each folder and website. Therefore I've used JSON files to store configuration (domain name, folder name, some rules for the program etc.). In total, I've got 650 JSON config files and the number is growing.

Now, I'd like to run the program in batches of 50 files. The members of one batch should be more or less 'static' (I want to manually delete one and add another for instance)

How would I "create batches"?

I can see several ways:

  • In the JSON file: "batch": 1, so it's hard coded in the config file
  • Create a separate database table / csv or similar like of key value pairs file1.json;1,file2.json;1 etc.
  • Using alphabet or divide the total amount of config files by batch size in a file iterator (where as, if I add config files, I can't influence in which batch a config file will be in this way)

Are there any other approaches/ideas on how to accomplish this?

  • 1
    All of these solutions can work, so there is no "right answer" to this question, and there are surely more variants (like making use of a folder structure with 50 config files per folder). This depends primarily on how you want to edit or control the groups of files, how they are processed, why and when you need to intervene manually - in short, your requirements, which only you know, not strangers from the internet. As a general hint: define more precisely what you want to achieve, what your goals are, then pick the most simple solution which fulfills this goals. – Doc Brown Nov 20 '19 at 7:12
  • I agree with you completely, therefore I've asked in the direction whether I've overlooked something. – Chris Nov 20 '19 at 20:58
  • From the "dont ask page" of the site's help center: avoid asking subjective questions where ... your answer is provided along with the question, and you expect more answers: “I use ______ for ______, what do you use?” – Doc Brown Nov 20 '19 at 21:47
1

All the solutions can work

And also, you have already pointed all the upsides and downsides

In the JSON file: "batch": 1, so it's hard coded in the config file

  • PROS: Ensures that all files will have a batch that it belongs to. (Forces coupling between files and batches)
  • CONS: Manually refactoring all 650 JSONS will be manual labor.

Create a separate database table / csv or similar like of key value pairs file1.json;1,file2.json;1 etc.

  • PROS: No need to manually edit all JSONS + granular control
  • CONS: Every time you add a file you will need to update this database/table. Bugs may happen if someone forgets to add this.

Using alphabet or divide the total amount of config files by batch size in a file iterator (where as, if I add config files, I can't influence in which batch a config file will be in this way)

  • PROS: You won't need to manually specify the batch for each and every file.
  • CONS: Does not allow custom batch control.

Evaluate if you are overthinking this.

There is a statement usually attribute to Kent Beck that I like very much:

Make It Work

Make It Right

Make It Fast

Perhaps having an easy way of moving files between batches is not yet a problem. Perhaps you would be better by choosing ANY of the options and waiting to see what problems arise with it.

It is also common to see the YAGNI (You Ain't Gonna Need It) principle be applied in this case. Do not write code "you might one day need it". Usually, that "one day" never arrives.

If anything, choosing the third option (implementing an arbitrary criteria based by filename) would probably be easy enough to implement and have batch working as fast as possible.

Of course, I do not know the bigger context of why you have those requirements - they could be very well justified. This needs to be your call.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.