I'm trying to create web app(flask or django-rest) that would scrape some data and save it to JSON so that it can be viewed in the frontend (VueJS).

I'm wondering if it is better to save the scraped data as JSON file in the database directly (noSQL?) and then create another api endpoint so that the frontend gets the JSON file raw from the database, or should I keep only scraped variables and values in the SQL database and then retrieve them to create JSON in the backend and then send it to the frontend.

What is better approach for this kind of app?

  • Sharing your research helps everyone. Tell us what you've tried and why it didn't meet your needs. This demonstrates that you've taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask – gnat Jun 19 at 8:03
  • So for me it feel like those two things are equivalent, as in it is possible for me to do both and I dont feel much difference, at least on paper. But maybe in software engineering world there are approaches considered "better" or more "correct", thats why Im asking here. – Alex T Jun 19 at 8:25
  • 3
    There's no such thing as "better" or "correct". These things are subjective. At least, they are conditioned by specific needs, requirements but overall, constraints. Note also that you can answer this yourself just by implementing some proof of concept – Laiv Jun 19 at 10:42
  • 1
    There's nothing wrong with storing JSON as a BLOB in a database – you really don't have to go NoSQL to do that. Better yet, nearly all SQL databases have native JSON support by now, meaning that you can write queries for JSON columns. Don't go NoSQL just because your data isn't very relational, only ditch classic DBs if they can't provide something you need. – amon Jun 19 at 17:31

The answer is (as usual) that is depends. If this data is going to be used for reporting and especially ad-hoc reporting, you probably want to pull the variables out into a proper relational model. Another reason might be if you need to expose that data in multiple formats. If you don't need to do any of that then the option of keeping the documents as-is becomes more appealing.

As amon notes, you can store JSON in blobs in a relational DB. The reason you might want to use NoSQL is scalability, speed, and potentially cost. If you already have a relational DB and you don't have a large amount of documents to store (e.g. millions) it might not be worth adding an additional DB. A lot depends on the context. If you are running on a cloud provider, spinning up a NoSQL DB for this might be less effort than getting the DB setup to hold the blobs.

| improve this answer | |
  • it might not be worth adding an additional DB. I would probably not be using any other DB at all. It seemed for me to be preference type of thing when it comes to choosing between NoSQL and SQL for all data storage. Or depending on your dev stack (MEAN or something else). – Alex T Jun 27 at 8:53
  • There are different benefits and trade-offs. A common refrain on this site is that using BLOBs and special non-relational features inside a relational DB is the same as using a NoSQL database. It's not true and the reality is that there are different flavors of NoSQL databases with very different capabilities. You really should think about it as an engineering problem, not a preference. – JimmyJames Jun 29 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.