2

I'm working on a project, I have a question regarding the architecture:

  • Say I have a many python scripts on my server and there's main.py which contains all the classes. And there's a script called copymain.py
  • A user named alex signs up and the url of his site is stored in mariadb
  • copymain.py checks if a user has signed up every 5 min using cron
  • when copymain.py detects that alex has signed up, it creates a copy of main.py rename it to alex.py, moves alex.py to scriptFolder, and writes the url of alex's site to alex.py
  • Cron or Celery (haven't decided yet) will run all the .py files inside scriptFolder every lets say 15 mins

Why I picked this design?

  • This design will let me get rid of many stack and queues and threading in my script and make it simpler, it is a simple solution, and it works perfectly. If I need to edit main.py, all I have to do is edit the modules imported, so I don't need to edit individual copies.

  • I think that by copying the files, I could easily deploy it on many servers. Move some .py files to this server, others to that and I'm done. These are the two main reasons.

  • Say you have to generate RSS for websites, for some reason there's a user called fred that has a problem, there's an issue that needs a special script, because we all know that every website has its own design and many errors occur when scrapping and dealing with html, you can go to fred.py and edit your script for that user.

Is it the most efficient architecture? Or should I use one file, get the users from database? My script currently does that, but I prefer to copy for the reasons stated above, a simple scheduling would do. I need to make sure that no matter how many users I have, the website of every user will be processed at exactly the time I promised when they signed up, when there's too many of them and I notice it's being slow, I just buy upgrade or buy another server. I'm afraid of creating too many threads and having to worry about it as it scales. Copying seems the simplest solution, Linux uses cron all the time for entire directories.

Edit

Please forget about the scrapping part, say you have to do some task for each user in your database, like sending him a message, or whatever, is my solution as good as queuing and threading, in terms of time spent on the process and in terms of being lightweight on the server? Is there a better solution that I have not considered?

Improve this question
9
  • 1
    So if I sign into your system, I get different behavior at 0–5 minutes and yet differenter behavior at 0–15 minutes? This seems like a horrible idea, in part because I've never seen users with auto-generated custom page generators that can be manually modified. Not only is it a version control nightmare, but it creates a huge vulnerability surface for security exploitation. I cannot understand your desires nor how these proposals improve anything. There exist well-established methods for any feature I can understand.
    – msw
    Jan 18, 2015 at 14:31
  • @msw lets say there are 10000 users and 1 million pages to be scrapped, i can't know for sure how long it will take to scrap them, meanwhile they may generate more feeds I might be interested in, because the queue would be so big and many websites to be scrapped, if I do it by copying, then I don't need to queue, whenever I find that cron is taking to much time, I add another cron at another time interval or buy another server. Besides the custom solution won't be available to everyone, say 10 clients at most, so I won't need to upgrade it much, just define some rules for their site
    – Lynob
    Jan 18, 2015 at 19:31
  • 1
    I cannot tell if you are trying to be protective of whatever your concept is or if you are not explaining yourself well. I do know that scraping 1 Mpages every 15 minutes or so is wildly inefficient for you and will tend to annoy their web administrators to the point where some of them are going to cut your scraper off. I suggest you look at scrapy.org in order to see how web scraping is handled conventionally.
    – msw
    Jan 18, 2015 at 20:17
  • @msw im scrapping only the rss page, and the web admin are the ones who apply for my service so if they don't want me to bother them they could unsubscribe from the service and I used to use scrapy in the past but it went down, happy to know they are working on the project again. My question has nothing to do with the scrapping, scrapping has its problem that I should solve, my question has to do with queuing and if as efficient or more than my solution in terms of time spent on the task given, and if there a better solution than queuing i'll be happy to know about
    – Lynob
    Jan 18, 2015 at 20:29
  • Your description is fairly imprecise. You'll get better answers if you go over it carefully while removing about 50% of the length. E.g., main.py contains the classes, yet it also doesn't contain actual code??
    – Dogweather
    Jan 20, 2015 at 8:17

3 Answers 3

Reset to default
2
+50

It's not uncommon to do what you described. In fact, when you create a new database for your user, you basically creating a new file for that user. So, it just adding a file to the set of per user files.

The choice of when to do the work begs explanation though. Doing the sign up process in batch periodically can create load spikes on the servers. It's better and simpler to copy all the files for new users straight away. If the process is long enough (e.g. creating a vm for user), you show them an "setup in progress" page.

Improve this answer
2
  • at least someone finds my solution not so awkward :D the files will be copied straight away, the scrapping will be done periodically though
    – Lynob
    Jan 21, 2015 at 0:30
  • no, not awkward at all. Pretty much like how virtual hosts are setup. Difference being shell scripts and you using python script. It's an Ops thing, I guess. With your scraping problem (which I think you should add to your question), you still have load issue I mentioned. One solution is to feed the jobs to a queue, like ZeroMQ or ActiveMQ and periodically check the queue, instead of going through users directories and/or database.
    – imel96
    Jan 21, 2015 at 1:08
3

Generally, this is not a good set up, because it's not DRY ("Don't Repeat Yourself"). If it were me, I'd work very hard to figure out how to have one well tested script which can generate the feeds needed by the individual users. Perhaps the script would get per-user information from the database.

Improve this answer
2
  • in terms of coding, I am following DRY, meaning my code is modular and I only need to edit one file and everything else will be edited, in terms of copying then I am not DRY :D I am wet :P but never read a book saying that copying is bad, they all say write modular codes and classes and write as little as possible, but no one say anything about copying :)
    – Lynob
    Jan 21, 2015 at 0:33
  • 1
    @Fischer: Behavior should be described once and only once in any well-designed application.
    – Kevin
    Jan 21, 2015 at 1:14
1

Disclaimer: I'm not entirely sure I understand what you're actually trying to accomplish here, so my answer is going to be somewhat vague.

Cron or Celery (haven't decided yet) will run all the .py files inside scriptFolder every lets say 15 mins

Let's be optimistic and suppose you have millions of users some day. Will Cron/Celery be able to get through all those scripts in a reasonable period of time? How long does each script take to run?

If your scripts are sufficiently complex, you will probably have difficulty scaling to millions of users. If they are not sufficiently complex, you can probably get by with a simpler design, perhaps using multiprocessing.

Improve this answer
5
  • most probably I'll use cron, and I'll move files to different servers in order to use cron efficiently, anyway what solution do you propose?
    – Lynob
    Jan 20, 2015 at 22:04
  • I'm not proposing a solution until I see a problem statement. You've talked a great deal about your architecture, but I don't understand what you're trying to do with it.
    – Kevin
    Jan 20, 2015 at 23:03
  • lets stick with the scrapping example, say you have to scrap websites for 1000 clients, each has 100 pages added per day, using python, so you opted to check for updates every 30 mins, how would you deal with the scheduling and scrapping?
    – Lynob
    Jan 21, 2015 at 0:23
  • one of the reasons i chose cron is because i can scrap 50 sites now, 50 at some other time interval, while if i were to use a database, the solution is harder to implement i assume, uses threads and queues which complicate the work and they will all run at the same time so if the scrapping is right now, then the server will take a hit
    – Lynob
    Jan 21, 2015 at 0:24
  • Databases exist to make things easier. For a problem such as this, your data is probably relational enough that a database will not make things significantly harder. Databases give you ACID, while filesystems do not. As for threads, they have nothing to do with databases. You can use threads, or databases, or both or neither, as you see fit.
    – Kevin
    Jan 21, 2015 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.