Storing scraping functions code inside a database

Question

We are building a system that scrapes data from many websites (curated list of 200+). Same data is scraped from all websites every fixed amount of time, but every website is required its own unique scraping function due to its DOM structure.

Creating a dedicated function for each website results in re-deploying the system for every change in one of these websites, not to mention code maintenance. We are thinking about storing the scraping functions inside the DB and using eval to execute them, so in case of a failure in one website due to out of date DOM structure, we can insert new scraping function for that specific website. We cannot simply store selectors, because of the significant variation among the websites.

We can really use some advice / guidance since we understand there is a discouragement with storing code inside a database.

Some tech details: JS-based platform + RDMBS (pg)

Answer 1

Although storing code in the database can work and can be made safe, I would seriously consider alternatives which allow you to put the code under source code control. Maybe changing your architecture so that scraping functions are loaded using some kind of plug-in mechanism would be an option? If each scraping function or class is in a separate file and is loaded on demand, you might be able to update it quickly without doing a full redeployment and without downtime.

Answer 2

Storing the scrapping function is totally fine as long as those functions are managed by the dev team, not from an end-user. However, it is hard to debug those functions, test them and track changes.

One of my favourite patterns to solve this kind of problem is using an event bus. To start scrapping a website, we dispatch an event with a specific topic (e.g. the website) with all input information. The scrapping function is implemented as event handlers, each function interested in a topic. Scrapping functions then register themselves to the event bus.

There are some advantages with this approach:

• Each scrapping function can be scaled separately, some websites may need more scrappers than the others.
• The logic of triggering the scraper and the scrapping function are decoupled. Scrapping functions don't know if they're triggered by a cron-job nor by a manual event.
• All code can be version controlled and deployed separately.
• With a good message broker (RabbitMQ, Kafka, ...) you can retry failed jobs.

Answer 3

There is really nothing wrong with storing a relatively short Python/Javascript/whatever program in a VARCHAR column, if that's what your problem could use. Maintaining interchangeable, fast-changing snippets of scraping functionality seems exactly the kind of use case that fits.

A lot of people will warn you about code injection vulnerabilities or the evaluator anti-pattern. Such problems are caused by someone lazily programming an eval() functionality into a program, expecting it to be used for one specific purpose, only to find later that attackers can abuse it to execute arbitrary code on the attacked system. That's a valid concern, and allowing execution of arbitrary code supplied from outside is almost always a big mistake.

However, in your case, evaluating code in a general programming language is the very point of your module, and presumably you treat the embedded programs as part of your code base and have them written, tested etc. by members of your team (and not by random Internet users). Therefore I think the reflexive "Don't program evaluation of arbitrary code!" you're going to hear a lot doesn't really apply here.

Answer 4

Just to give you a data point for a software project that does store scraping code for hundreds of webpages: youtube-dl, a Python library (and command line tool) to download videos from a large number of video streaming platforms.

The large list of extractors is kept under source control, to allow for streamlined maintenance of these modules. However, their deployment situation (creating a new release) is probably much less time-critical than your usage scenario.

Answer 5

If it were me, I'd write all scraping functions to be stored in one repo and forget about the DB with respect to each web site. Why create different code for what is almost certainly already a code pattern to be contained in the root solution?

Say hello to Generic Types as they are the key to consolidating common patterns but applying them to unique situations. Also look into Code Generation of models that are specific to those sites, in that case you could use a db to store the models for each site, but again how many models are being created that would fit into a more common Generic model? The answer is you don't know until you've done extensive analysis of what's been generated.

Finally the concept of autogenerated code really means you don't have to store what it produces; after all it generates anew each time. Look at the autogenerated code as just a part of the total picture, whereby an external controller uses the generated code. That way if the controller fails after a new generation you find a potential bug or perhaps the need to change the controller and not the generated code. Keep this up for a year or two and you'll automatically know the common patterns which are controlled by the code generators in conjunction with the controllers which now just work.