We are building a system that scrapes data from many websites (curated list of 200+). Same data is scraped from all websites every fixed amount of time, but every website is required its own unique scraping function due to its DOM structure.
Creating a dedicated function for each website results in re-deploying the system for every change in one of these websites, not to mention code maintenance. We are thinking about storing the scraping functions inside the DB and using eval to execute them, so in case of a failure in one website due to out of date DOM structure, we can insert new scraping function for that specific website. We cannot simply store selectors, because of the significant variation among the websites.
We can really use some advice / guidance since we understand there is a discouragement with storing code inside a database.
Some tech details: JS-based platform + RDMBS (pg)