A program design question: Good idea using HDFS in c for reading large data?

Question

I have mainly three groups of CSV files (each file is divided into several small files): First group of CSV files have 600+ GB in total (MAYBE 200+ GB if in int, cause CSV calculates by char right?), each file has same size of lines, the whole data should be gather line by line together. The application should read some specific lines from these files, gets the data_1 and fetches the data_2 and data_3 from second and third group of files by data_1. Second and third group of CSV files has about 60 GB each. Add together the lines of all files in order, then it would be the whole data.

Here comes my question, it costs such a long time to reads some specific lines from the three groups of files, where I use fgets() in c to read CSV file. I got a large RAM (about 190GB), but not enough to load all the files of first group, I tried PSQL as well, it works much better than reads from CSV files. But I am wondering, if there is any other way to make it perform better? if HDFS is a good idea?

Answer 1

For only reading 10 MB of data out of a 600 GB file, you don't need a distributed filesystem, you need a way of reading only what you're interested in.

Databases with indexes come to mind or even performing binary search on the file if it doesn't change often (requires the rows to be sorted by the search key and rows of equal size so you can quickly jump to a specific row without iterating from the begining every time).