How to batch organize configuration files?

Question

I've got a program for analyzing specific folders / websites. The program always does the same steps for each folder and website. Therefore I've used JSON files to store configuration (domain name, folder name, some rules for the program etc.). In total, I've got 650 JSON config files and the number is growing.

Now, I'd like to run the program in batches of 50 files. The members of one batch should be more or less 'static' (I want to manually delete one and add another for instance)

How would I "create batches"?

I can see several ways:

• In the JSON file: "batch": 1, so it's hard coded in the config file
• Create a separate database table / csv or similar like of key value pairs file1.json;1,file2.json;1 etc.
• Using alphabet or divide the total amount of config files by batch size in a file iterator (where as, if I add config files, I can't influence in which batch a config file will be in this way)

Are there any other approaches/ideas on how to accomplish this?

Answer 1

All the solutions can work

And also, you have already pointed all the upsides and downsides

In the JSON file: "batch": 1, so it's hard coded in the config file

• PROS: Ensures that all files will have a batch that it belongs to. (Forces coupling between files and batches)
• CONS: Manually refactoring all 650 JSONS will be manual labor.

Create a separate database table / csv or similar like of key value pairs file1.json;1,file2.json;1 etc.

• PROS: No need to manually edit all JSONS + granular control
• CONS: Every time you add a file you will need to update this database/table. Bugs may happen if someone forgets to add this.

Using alphabet or divide the total amount of config files by batch size in a file iterator (where as, if I add config files, I can't influence in which batch a config file will be in this way)

• PROS: You won't need to manually specify the batch for each and every file.
• CONS: Does not allow custom batch control.

Evaluate if you are overthinking this.

There is a statement usually attribute to Kent Beck that I like very much:

Make It Work

Make It Right

Make It Fast

Perhaps having an easy way of moving files between batches is not yet a problem. Perhaps you would be better by choosing ANY of the options and waiting to see what problems arise with it.

It is also common to see the YAGNI (You Ain't Gonna Need It) principle be applied in this case. Do not write code "you might one day need it". Usually, that "one day" never arrives.

If anything, choosing the third option (implementing an arbitrary criteria based by filename) would probably be easy enough to implement and have batch working as fast as possible.

Of course, I do not know the bigger context of why you have those requirements - they could be very well justified. This needs to be your call.