0

I am creating a DSL for a scraping library I am writing. I would like advice on how to design a DSL, and if the designs I have below are good ones. Apologies if this is an open-ended question, but it appears that this stack is a good place to get advice, so here I am.

The npm package in question is called scrape-pages. The basic idea is to avoid writing code when writing a scraper. Instead, scraper instructions will be stored in json format. The package works already, but I am constantly flip-flopping on what the config structure should look like. It is essentially a DSL. A good senior programmer once taught me that when designing a DSL, it is the data structure that matters, not the syntax. So I have been trying to focus on a clear and concise config structure.

The current design looks like this:

// I am hiding DownloadConfig and ParseConfig, since they only store flat data
// specific to downloading and parsing at each stage. E.g. urls, css selectors, ..

type ScraperName = string
interface ScraperConfig {
  download?: DownloadConfig
  parse?: ParseConfig
  incrementUntil?: 'failed-download' | 'empty-parse' | number
}
interface Structure {
  scraper: ScraperName
  forEach: Structure[]
  forNext: Structure[]
}
interface Config {
  input: string[]
  scrapers: { [scraperName: string]: ScraperConfig }
  run: Structure
}

I also have a competing design that flattens the structure somewhat.

type ScraperName = string
interface Scraper {
  name: ScraperName
  download?: DownloadConfig
  parse?: ParseConfig
  incrementUntil?: 'failed-download' | 'empty-parse' | number
}
interface FlowStep {
  scrape: Scraper
  branch: FlowStep[][] // similar to forEach
  recurse: FlowStep[][] // similar to forNext
}
interface Config {
  input: string[]
  flow: FlowStep[]
}

I do not know which of these is better. I have been trying to work with a few core assumptions about what the config needs to do, and use those to help me design it, but I am not sure if these are good assumptions to begin with:

  • each scraper will output an array of values, so the design needs to represent a flow of data
  • the config needs to support branching (e.g. one input can be passed to multiple scrapers
  • the config needs to support 'recursion' (e.g. a value from a scraper can be fed back into the same scraper)

Are these good guiding assumptions? Is there another good core aspect I can use to guide which design to pick? I'd like to work with some absolutes that help me when making big decisions like this, but because the package is so new, I can change whatever I want. That is great, except I never know when to stop refactoring.

[note] for each of these Config types, I have a corresponding ConfigInit type which is what the user will actually input. They essentially remove a bit of the verbosity of these configs (essentially where there is an empty array, you can just skip writing that key). I didnt show them so it would not clutter this question, but I can edit the question if it is helpful. I can also provide example usage of the configs if that is helpful.

  • I cannot tell you which design is "better", but if I were in your shoes, I would probably find some web sites where you can test your DSL and gain some real-world experience (maybe with both approaches, so you can compare them). – Doc Brown Sep 9 at 6:23
  • ..somewhat off topic, but I used to write crawlers for fun. I think crawling and parsing HTML won't work anymore since most pages will have content constructed using javascript frameworks. For making a crawler in 2019 I'd look into using something like puppeteer. – C.M. Sep 9 at 13:16
  • @C.M. thats only true for sites that are true SPAs (e.g. they only have client side routing). Most modern sites that use client side routing still use server side routing for initial page load. It makes for a better user interaction. Of course, there are more barriers than there used to be (SPAs, "Are You A Robot", encryption) but there is still a significant amount of sites that can be scraped for me to feel its worthwhile. Oof you got me off topic too! – andykais Sep 9 at 13:36
0

I think the essential requirement for a scraper is flexibility and extensibility. When I think about "Scraping" my definition is to extract certain information from some data sources.

The "common" case is to save the RAW html of a certain "starting" URL, traverse ALL the links on the page, queue these pages to be scanned too, and then repeat.

But talking about flexibility, you can differentiate between certain steps here:

  • Retrieving the data from the source (e.g. simply grab URL contents)
  • Deciding how to handle the data source contents
  • Parsing the source data to extract relevant information (e.g. extract raw html)
  • Parsing the source data to know what other data to parse (e.g. extract links)

Thinking about different use cases I find that you will might need different ways of retrieving data, e.g. just grab URL contents will not work on dynamic pages - you might need to login somewhere first, or use a DOM based approach to handle dynamic pages, etc.

For extracting the information from the page, saving the raw html is one option, but maybe you want only a part or extract certain other information from the page. Different ways to implement here too.

Same is true for checking which other "pages" to scrape. Somebody may want all links, or only the links inside the page, or only links that contain a certain regexp, or ...

Your "data model" should somehow reflect this variability. It is without an explanation of your "scraping" algorithm and especially your goals for the scraper hard to give a good advice how to design the data model of your DSL. But to allow extension, I would expect at least, that there are "different" components requiring possibly different parameters. While a "link extractor" that extracts links based on a regexp requires a regular expression, another link extractor will need a different configuration. This variability will need to be reflected on your data model.

At least if variability is a goal at all for your scraper.

  • Hey Mathias, the package actually does support retrieving data in different ways. If you look at the README of the project you will see that data is grabbed, and a parsing format can be chosen, (e.g. for an html page use css selectors to find links/data). The "algorithm" is actually the part I am trying to decide with this question. What I want help with, with this question, is describing the data flow model, not the actual download/parse steps. I have those covered. I think most of your questions can be better answered in my readme than here. – andykais Sep 9 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.