I have mainly three groups of CSV files (each file is divided into several small files): First group of CSV files have 600+ GB in total (MAYBE 200+ GB if in int, cause CSV calculates by char right?), each file has same size of lines, the whole data should be gather line by line together. The application should read some specific lines from these files, gets the data_1 and fetches the data_2 and data_3 from second and third group of files by data_1. Second and third group of CSV files has about 60 GB each. Add together the lines of all files in order, then it would be the whole data.
Here comes my question, it costs such a long time to reads some specific lines from the three groups of files, where I use fgets() in c to read CSV file. I got a large RAM (about 190GB), but not enough to load all the files of first group, I tried PSQL as well, it works much better than reads from CSV files. But I am wondering, if there is any other way to make it perform better? if HDFS is a good idea?