Is it good style to have a class that only stores data for abstraction?

Question

I have written a webscraper in Python that grabs data from ~10 different websites (they have confirmed that this is ok). Most of the websites are a little bit different, but the general idea of getting data from each is the same:

1. Load the site's URL
2. Press anywhere from 0 to 3 buttons (depending on the site).
3. Enter a site-specific search phrase.
4. Grab data from any result which contains yet another site specific phrase.
5. Record the data in a file, specific to the site.

Obviously, most of this can be abstracted away, but steps 3 and 4 are too different between sites to abstract the whole process. So there is an abstract scraper, extended by a site-specific scraper for each site which handles steps 3 and 4.

My problem is that there's a lot of site-specific information to keep track of. I could put each site-specific configuration in its own class, but I'd like to keep it all in one place so I can make mass changes (digging through 10 classes to make the same change sounds terrible).

What I did was create a class I called "ScraperData", which looks like this:

class ScraperData:

    scrapers = {
        "Website1": Website1Scraper(),
        "Website2": Website2Scraper(),
         ...
    }

    URLs = {
        "Website1": "www.website1.com",
        "Website2": "www.website2.com",
        ...
    }

    buttonsToPress = {
        "Website1": None,
        "Website2": ["Button1", "Button2"],
        ...
    }

    ....

I know I could achieve something a bit less wordy by restructuring this (I could have a configuration class and initialize one for each site in here), but I like that this approach lumps like data together.

I have a config file as well which has one line:

Website1, Website2, Website5, ...

Where I pick which sites to get data from (we don't always want to do all 10).

Lastly, there is a main class which loads the websites from the config into an array, sitesToScrape, and does roughly the following:

data = ScraperData()

for site in sitesToScrape:
    scraper = data.scrapers[site]

    scraper.load(data.URLs[site], data.buttonsToPress[site])
    scrapedData = scraper.scrape(data.searchStrings[site])

    self.write(site, data.files[site], scrapedData)

Each morning I just pick which sites to grab data of off, start the whole thing up, then leave it alone for a few hours.

The way I see it, this makes it really easy to plug in new sites, take out old ones, or edit the configuration of one or multiple sites. The whole project feels weirdly compartmentalized though, but I feel like each compartment serves one specific purpose.

My question is: Is this an easily maintainable approach to this problem? Performance is not a factor at all.

Answer 1

For what it's worth, I did a lot of work exactly like this at a previous job. Ten classes doesn't seem like a lot to me.

We dealt with hundreds of websites. Originally, we stored the configuration data for the scrapers in a database, but since the original developer never pulled the trigger on a configuration UI, maintenance became a nightmare. The base class for the scrapers was some 1000 lines of code, and populating the configuration information required hundreds of lines of SQL script for each scraper.

Eventually, I ruthlessly refactored this code until I could write a scraper in about 30 to 100 lines of code in a couple of hours, and I incorporated the configuration information into each scraper class, directly. In practice, what you're going to find is that you will be changing the code and the configuration for each scraper at the same time, so it's better from a maintenance standpoint to have it all in the same place, since the configuration information for any given scraper will never be used anywhere else.

We found that websites are different enough from each other that trying to find commonalities to put in a base class or configuration file simply isn't worth it. Instead, what you want in your base class is robust cookie and view-state handling, and dead-simple support for HTTP GET and POST operations.

That said, our overall approach to this problem was rather novel. These small scraper classes ran in an Actor Model environment, where workflows could be defined and the scraper code could be hot-swapped.

But is this approach scalable? Is it flexible? Is it maintainable? Well, let's see: I was able to write 50 scrapers for 50 different websites over a period of about six weeks (slightly less than two per day). The Actor Model that they ran on is capable of being scaled up just by adding additional servers and registering them with the database server.