I currently own a website where players can look-up their statistics of a game. When someone that is not in my database yet requests their statistics, I am calling the API with a PHP cURL request and storing them in my database.
Then I have a cronjob that runs every minute, takes 1/5th of the players in the database and updates every user in the database with the external API (20 users each request).
This worked fine when my mysql database was small (1000/2000 users) but the amount of users is increasing really fast and will become a lot more soon, so I need to change how I am updating the users.
My cronjob currently calls a php script that sends cURL requests and updates the database with the response.
Would it be better to update the players every hour and change the php cronjob to a python cronjob? I could call the API when the user requests their statistics but the external API is very slow sometimes or does not respond and this won't allow me to create graphs.
Looking forward to some thoughts about this.