0

Consider the following application structure.

enter image description here

The scraper class extracts the necessary information from a page, and I want to save it to the DB.

The ORM class is a wrapper around sequelize.js. It establish a connection to the db, loads the models, synchronizes with the db.

app.js inits the ORM, then the Scraper class.

My question is: how should the scraper class flush the collected data to the db? It should import the ORM (the connection) and have a flush method, which checks, if the data was already scraped. If not, then saves it. Something tells me, it should not be the job of a scraper class.

1

You are missing an intermediate class that collaborates between the Scraper and ORM (or Persistence Layer).

Many technology stacks call this sort of thing a "service," which I have a great disdain for, because it doesn't describe what it actually does.

What you need is some sort of "ScraperJob" which is the marriage of the two layers of the application. It would:

  • Take any configuration data necessary to "scrape" something
  • Take a reference to the "ORM class"
  • Do the scraping
  • Contain logic to transfer data from the Scraper class to the ORM class
  • Contain the transaction logic for the ORM, such as telling the ORM when to make a commit to the database or file system

Updates from comments:

Should the scraper/scraper.js be responsible for this? and scraper/scrapeSiteA.js and scraper/scrapeSiteB.js should make the actual scraping?

If scraping sites cannot be abstracted, then having one scraper class for each site would be ideal. Your idea about naming the scrapers is correct in this case. You would want each concrete scraper to implement the same interface (or public methods, since this is JavaScript). That way you can deal with each site as an abstraction by utilizing some basic OOP features:

class Scraper {
    constructor(url, yourOrm) {
        this.url = url;
        this.yourOrm = yourOrm;
    }

    getPageContents() {
        // TODO: Make request, return response body
    }

    scrape() {
        throw new Error("Child classes must override this method");
    }
}

class MovieRatingSiteScraper extends Scraper {
    scrape() {
        // TODO: Write me
    }
}

class DiningGuideSiteScraper extends Scraper {
    scrape() {
        // TODO: Write me
    }
}

And to use it:

let yourOrm = // some object that represents your "ORM"
let scraper1 = new DiningGuideSiteScraper(yourOrm, "http://somediningguide.com");
let scraper2 = new MovieRatingSiteScraper(yourOrm, "http://somemoviewebsite.com");

scraper1.scrape();
scraper2.scrape();

With this level of control, you have the option of tailoring each scraper not just to an entire site, but specific pages on each site - giving you very fine grained control of this process, as well as any specialized error handling you want.

  • I knew a chain (layer) is missing. What name would you recommend, and where would it fit int the structure? Should the scraper/scraper.js be responsible for this? and scraper/scrapeSiteA.js and scraper/scrapeSiteB.js should make the actual scraping? – user3568719 Jul 10 '17 at 13:06
  • Thanks the edit of the answer. Where would you put the ScraperJob (as you called it) ? I guess the that class would do, what you gave as example after the '>And to use it:' part. I think it also should do the last two bullet points. ('Contain logic to transfer data from the Scraper class to the ORM class' and 'Contain the transaction logic for the ORM, such as telling the ORM when to make a commit to the database or file system'), do you agree? – user3568719 Jul 10 '17 at 19:16
  • I would put the scraper job in the "scraper/" folder. And you are correct. The last two bullet points would go in the specific scraper job. – Greg Burghardt Jul 10 '17 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.