0

I have mainly three groups of CSV files (each file is divided into several small files): First group of CSV files have 600+ GB in total (MAYBE 200+ GB if in int, cause CSV calculates by char right?), each file has same size of lines, the whole data should be gather line by line together. The application should read some specific lines from these files, gets the data_1 and fetches the data_2 and data_3 from second and third group of files by data_1. Second and third group of CSV files has about 60 GB each. Add together the lines of all files in order, then it would be the whole data.

Here comes my question, it costs such a long time to reads some specific lines from the three groups of files, where I use fgets() in c to read CSV file. I got a large RAM (about 190GB), but not enough to load all the files of first group, I tried PSQL as well, it works much better than reads from CSV files. But I am wondering, if there is any other way to make it perform better? if HDFS is a good idea?

8
  • How much data to you normally need to read from the 600GB? – D. Jurcau Mar 10 '20 at 16:49
  • 2
    There's a good chance that you don't need all of the data in memory at the same time to solve whatever problem you're trying to solve. – Robert Harvey Mar 10 '20 at 18:20
  • like 10 MB or even less.. – heisthere Mar 10 '20 at 18:58
  • @RobertHarvey The challenge is that the application have to be really fast, but the most direct way (load all data in memory) does not work, so I am trying to figure out somehow... – heisthere Mar 10 '20 at 19:00
  • 1
    PSQL works faster because it is indexed allowing you to zero in on your data faster. At that point, you only need something to load your files into your database, which should be relatively quick--even if you use a higher level language. – Berin Loritsch Mar 10 '20 at 20:54
2

For only reading 10 MB of data out of a 600 GB file, you don't need a distributed filesystem, you need a way of reading only what you're interested in.

Databases with indexes come to mind or even performing binary search on the file if it doesn't change often (requires the rows to be sorted by the search key and rows of equal size so you can quickly jump to a specific row without iterating from the begining every time).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.