0

I'm writing a system where data will be extracted from HTML formatted strings.

Different types of strings will contain the same type of data, but the methods to extract that data will vary between the types of strings that are being parsed.

Ideally I'd like to be able to store the function to extract the relevant data somehow, rather than hard code it. I haven't worked on a system that's implemented anything like that before, though, and I wonder about a few potential downsides:

  • Security risks: running code from a database does not seem secure. My best thought would be to use some data storage format which can be parsed to extract the relevant data (ie. like a regex string only more complicated)

  • Complexity: it'd be very difficult to write a regex string to be able to parse these strings, but it's presumably techincally possible - my point is, is any way of storing these queries other than source code likely to be very difficult to write maintain

  • Novelty: there might be good reasons (see above) why this isn't something I've come across before, or it might be something that is done, but I just haven't seen yet. If such a way of encoding string parsing exists it'd seem to make sense to use that first

The short version of this question could be "is it a good idea to store string parsing functions in a database"


I thought it might help to include an example of the type of thing I'm thinking. If we wanted to extract the price of a product from the following string:

<h1>Chocolate bar</h1><p id="price">£0.59</p>

We might encode some array of selectors:

[
   {
      "type":"selector",
      "payload":"p"
   },
   {
      "type": "extract-text"
   },
   {
      "type":"replace",
      "payload":{
         "a": "£",
         "b": ""
      }
   },
   {
      "type": "to-number"
   }
]

In this way the type of parsing can be controlled and restricted, but stored in a general way such that any given valid HTML string can be parsed to extract the relevant data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.