I'm working on a project, I have a question regarding the architecture:
- Say I have a many python scripts on my server and there's main.py
which contains all the classes. And there's a script called
- A user named alex signs up and the url of his site is stored in mariadb
copymain.pychecks if a user has signed up every 5 min using cron
copymain.pydetects that alex has signed up, it creates a copy of
main.pyrename it to
scriptFolder, and writes the url of alex's site to
- Cron or Celery (haven't decided yet) will run all the
scriptFolderevery lets say 15 mins
Why I picked this design?
This design will let me get rid of many stack and queues and threading in my script and make it simpler, it is a simple solution, and it works perfectly. If I need to edit
main.py, all I have to do is edit the modules imported, so I don't need to edit individual copies.
I think that by copying the files, I could easily deploy it on many servers. Move some
.pyfiles to this server, others to that and I'm done. These are the two main reasons.
Say you have to generate RSS for websites, for some reason there's a user called fred that has a problem, there's an issue that needs a special script, because we all know that every website has its own design and many errors occur when scrapping and dealing with html, you can go to
fred.pyand edit your script for that user.
Is it the most efficient architecture? Or should I use one file, get the users from database? My script currently does that, but I prefer to copy for the reasons stated above, a simple scheduling would do. I need to make sure that no matter how many users I have, the website of every user will be processed at exactly the time I promised when they signed up, when there's too many of them and I notice it's being slow, I just buy upgrade or buy another server. I'm afraid of creating too many threads and having to worry about it as it scales. Copying seems the simplest solution, Linux uses cron all the time for entire directories.
Please forget about the scrapping part, say you have to do some task for each user in your database, like sending him a message, or whatever, is my solution as good as queuing and threading, in terms of time spent on the process and in terms of being lightweight on the server? Is there a better solution that I have not considered?