I've got a program for analyzing specific folders / websites. The program always does the same steps for each folder and website. Therefore I've used JSON files to store configuration (domain name, folder name, some rules for the program etc.). In total, I've got 650 JSON config files and the number is growing.
Now, I'd like to run the program in batches of 50 files. The members of one batch should be more or less 'static' (I want to manually delete one and add another for instance)
How would I "create batches"?
I can see several ways:
- In the JSON file:
"batch": 1, so it's hard coded in the config file
- Create a separate database table / csv or similar like of key value pairs
- Using alphabet or divide the total amount of config files by batch size in a file iterator (where as, if I add config files, I can't influence in which batch a config file will be in this way)
Are there any other approaches/ideas on how to accomplish this?