Looking for best practices of storing configurations that will be read/write frequently

Question

I'm writing a couple small python scripts to do tasks and act as 'bots' in some applications that will all read/write things for configuration and some very low-level logging of information; Whats the best way of storing these snippets of information?

Currently I've made a single config.json file that has Bot tokens, user information with ID numbers and configurations of the bots such as if theyre enabled or not, file names and when they last produced a response.

The bots will be running in nested screens on a Raspberry Pi I have, so my plan was to have them update the JSON file for their relevant "Last Response" key so that my 'controller' script can read this and alert me if they have not updated in [x] minutes based on their frequency of running, I will be checking if the screen is still active but dont know how to read into that screen to see if it is still printing.

The concern I have is having multiple files write to the same JSON can conflict and cause an error if not handled and not write the value as expected, the work around would be smaller individual config files or logging files to record when they responded and then the controller can gather that information to prevent multiple writes from different sources.

Is there a better way of doing this than I already am or could do?

Answer 1

Personally, I would investigate a solution using SQLite.

From the SQLite FAQ:

Can multiple applications or multiple instances of the same application access a single database file at the same time?

Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.

Each bot can get it's own row in one or more tables and can retrieve or update its settings as needed.

Answer 2

This description seems little bit complicated to me. But as I understand you're running small snippet bots on pi.

Pi has limited processing power and RAM compared to cloud machines. That's why I am not going to suggest you to use a database. See if there is any light weight cache that you can use on pi as a hack around. It can make your life easier.

Another option is to use system environment variables (at Linux level) to store some configuration. It will be useful only to check which bots are enabled or not (kind of true false values) and some short string storage, you should definitely not use it to store logs.

Both options I have mentioned feel like hack around to me at certain level. But if it can alleviate it's good enough for a small application :)

Answer 3

Another solution is to separate control and configuration from the execution: Do the execution logic on the PI and let the bots retrieve any config from a server application running somewhere else; resp. put information from the bots to this server.

What you do there is independent from the PI.