1

I am developing a mobile app that depends on some nearly static data.

I will gather that data initially from an existing website by doing some HTML parsing.

I could now create a database from that data and use this as my backend for the app. Whenever there is new data, I will manually run my parsing program again on the website and add the new items to my database. But since I am parsing all the time anyway I thought I maybe could use the website itself as the backend, i.e. instead of querying a database, I could run the parsing directly.

What do you think? Will this work or is it too unstable?

Thanks in advance!

Improve this question
1
  • Too unstable for what? If the website changes to a point where it breaks the parsing (You have to determine the level of risk), will users really care? How much of a delay can they live with?
    – JeffO
    May 14, 2018 at 15:09

1 Answer 1

Reset to default
8

I strongly recommend using a database with a well-defined schema. Why? Independence/Decoupling. With a database, things are under your control, not theirs. Lets look at the pros and cons of each approach:

live parsing

  • Pro: Always up-to-date
  • Con: Availability depends on the source website
  • Con: Performance/Speed depend on the source website
  • Con: If the source website change their format, your site is down. Immediately.
  • Con: About twice as much traffic from&to your server that you'll have to pay for

Database

  • Con: Not always up-to-date, delay of updates is hard to get below 1 minute
  • Pro: You're independent. You're in a good position if
    • The source site is down (your site will stay up)
    • The source site changes its format (your site stays up and you can unhurriedly adapt your code
    • Speed depends on the scaling of your servers, not theirs.
    • Pro: If you decide to gather your data from somewhere else, the transition is way more smooth and way less work for you.

Also, consider the legal aspect of this. Make sure you are allowed to use the data of your source. If they decide to lock you out or sue you out of using their data, you need to be flexible. With a database, you have that flexibility. With live parsing you'd have that flexibility, too, but it comes with more work and less stability of your own service.

Last, but not least, the database has two important advantages live parsing cannot possibly give you: you can set your schema up so you can

  • track the history of the data and show histories and statistics
  • correct/adjust certain data points that you deem incorrect
Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.