0

I plan on using Scrapy to crawl a local website for a LOT of data and store it in a file. Then I plan to parse that file and put some of the data in a SQL database.

Will my computer use less CPU and RAM to read and parse a big CSV file or JSON file?

Or maybe it would make more sense to store the data in a bunch of smaller CSV or JSON files?

Either way, which is less taxing on my machine?

|improve this question
  • 9
    Why don't you measure? – πάντα ῥεῖ Mar 25 '18 at 19:33
  • 3
    How often will you need to read the JSON/CSV file? If the answer is "not often", then the answer is "relative to the scraping, this time is trivial". – user949300 Mar 25 '18 at 19:41
  • 7
    What prevents you from writing the data into the database directly? – Robert Harvey♦ Mar 25 '18 at 20:02
1

Parsing a JSON file is more complex than a CSV file.

In JSON you have to deal with {}, [], ":", "," and extra complexity with nestedness. In CSV you only deal with line breaks and colum separators.

Because of CSV's simplicity, you can do chunkwise reading (streaming) much easier, so if your file size is going to be greater than a few gigs (like > 4gb), the reading logic will be much simpler and more efficient for CSV. In such a case you will be forced to do chunk-wise reading because you will not be able to load up the entire file into RAM...

So overall, I would say - go with CSV.

|improve this answer
  • 5
    "In CSV you only deal with line breaks and colum separators." - LOL if only life was that simple. – whatsisname Mar 25 '18 at 21:01
  • Mmhhh... no. Parsing JSON is easier (json.load ()), and I don't buy the streaming argument. CSV is as broken as this question and your advice – Tibo Mar 25 '18 at 21:04
  • You are assuming the parsed data itself won't include line breaks or the same characters you are using for column separators. Things get messy. – Shafiq al-Shaar Mar 25 '18 at 21:20
  • @Tibo The fact that some library hides the complexity of the json parsing, doesn't make the complexity vanish. There's a lot more CPU and memory management activity to be done for parsing the json (especially to be able to cope with its dynamic structure and with embedded variable sized containers) than for a flat csv file. – Christophe Mar 25 '18 at 21:28
  • 2
    @Tibo: that's not the point. The point is you can't judge the complexity of something based on how many arguments a function call takes. – whatsisname Mar 26 '18 at 1:23
-1

What a dilemma!

  • On one hand CSV is never good.

  • On the other, a flat file format is always going to be 'cheaper' than a nested one.

JSON at least has a spec, even though it has problems with basic things like numbers.

I think the correct answer is that if your data structure is simple and well defined, you can probably hand craft a super slimline csv-style parser which is more performant than an off the shelf JSON one.

The minute your data gets at all complex, ie escape characters, second level delimiters, header rows, long data in columns etc. Its questionable whether this 'extended csv' structure is any less complex and hence faster to parse than a nested format such as JSON or XML.

Regarding the bigger questions raised by your post.

  • Writing files of any sort is never 'fast',
  • You would have to be talking gigabytes of data before you noticed a difference
  • Writing parsers at this level is probably more complex than scraping the data. Just test some off the shelf ones and chose the fastest
  • Databases are specifically designed for this kind of thing. Use them!
|improve this answer

protected by gnat Mar 29 '18 at 18:16

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.