1

I'm constantly supporting a specialized downloader I wrote 5 years ago! It has to login and download from more then 40 sites. I have kept changing the code over and over as I encountered new sites or when a site's structure completely changed.

This approached worked until now, but recently more and more changes are coming my way and it has become very hard to implant them, my main class is almost 5k lines and I was wondering:

  1. Should I continue this approach and soldier through and fix what needs to be fixed?
  2. Should I rewrite the whole class which might take at least 2 weeks which I will not get paid for! and will force my clients to wait for a while!
  3. Or should I try creating a new class for each new site that comes, this class can inherit the main class and I can overwrite any function which needs to be overwritten for this special site.

Personally, I'm thinking option 3, any advice?

Improve this question
2
  • option 4 hire junior devs to do the work?
    – Ewan
    Commented Feb 27, 2019 at 16:07
  • @Ewan The pay for the support is not bad but its definitely not high enough to hire someone! plus this is a free lance work which usually i had to do nothing for it, sometimes even for months! but as i said recently too many sites broke and it got me thinking that i should probably change my code's structure.
    – Ali Ahmadi
    Commented Feb 27, 2019 at 16:11

4 Answers 4

Reset to default
3

It sounds like you are in a competition with the sites you are scraping. ie. They haven't signed up to your service and don't care if it breaks.

In this situation you can never design a generic solution which will cover all the specific cases you have to deal with. Refactoring and spliting out the code as much as possible is a good idea, but you will always have a constant stream of customisation and updating work.

I would advise not trying to create a generic solution. Split the code into multiple custom solutions

Improve this answer
3

Separate things that change from things that stay the same.

Make a specialized class for every site. Make generic classes that cover most needs. Point each specialized class at the most generic class that does what it needs. Needs that can't be met generically can be met with code in the specialized class.

This keeps you from exploding the code but also lets you customize as needed.

Do this and when change happens you won't be forced to watch it spread any further than is absolutely needed.

Improve this answer
2
  • If possible, don't make a different class, but one class that takes some configuration, and use different configurations. Sites that cannot be handled that way get their own class.
    – gnasher729
    Commented Mar 2, 2019 at 11:31
  • @gnasher729 that can work as well but it's preferable if somewhere there exists an explicit statement of how a site is handled. It's just more work. Either way when the catch all works point the site to the catch all.
    – candied_orange
    Commented Mar 2, 2019 at 16:57
2

I have made some web crawler for work and know the pain of site redesigns. the solution I found was a mix between option 3 and option 2.

Make a new base with the bare minimum of shared code and make a new class for each new site with all the specialized code in it, meanwhile try to rewrite your older sites into new classes one at a time. You won't stop providing a service nor denying new sites to your clients but eventually you'll finish migrating the whole code to a better architecture.

Improve this answer
0

Option 3 makes a lot of sense regarding maintainability. You might also consider option 4: Implement a rule-driven engine, where the rules are used to specify what to do when. You might even be able to have the software update its rulesets from your site (like AV programs do) and deploy actual software versions less often.

Improve this answer
1
  • Actually the main class does kinda work similar to a rule-driven engine, because i have a configuration file beside my exe, which each site has a setting file inside of it and that setting for example has things like "the login URL=x" or "isCaptchaEnabled=false" and many other options which actually worked great at the beginning but the main problem was that the rules became too specific as time went on and i kinda forgot some of them and i had to read the code to understand whats going on!
    – Ali Ahmadi
    Commented Feb 27, 2019 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.