I have a method for reading data from file. The problem is how to handle files that are too big for a simple read and save to database? I was thinking about reading a chunk of it and saving it to database, but I don't know if having an asynchronous method with callback is a good idea.

Basically I think that a reader class shouldn't be aware of any database interface, so in order to notify of successfully reading a chunk of data it has to have a callback. I don't know if this is a good approach or not.

private const int Buffer = 100000;

    public Task ReadAsync(Action<Tuple<DataTable, int>> statusCallback) {
        DataTable data;
        return await Task.Run(() => {
            var totalRows = GetRowsCount(); // iterates file to calculate total number of rows
            var progress = 0;
            if(totalRows < Buffer) {
                /**Read whole file...*/
              progress = 100;
            else {
              while(/**loop until end of file*/) {
                   for(var rowIndex = 0; rowIndex < Buffer; rowIndex++) { 
                      var row = reader.Read();
                      /**Split, parse, etc...*/
                progress += Buffer/totalRows * 100; // Add read rows to total result %
                statusCallback(new Tuple(data, progress));


And then save it

public void Start() {

private void ReadingProgress(DataTable data, int progress) {
     using(var tran = _database.BeginTrans()) {
         foreach(var row in data.Rows) 

For some reason it seems wrong to me, but I don't know why. Any ideas how I could improve this?

EDIT: I would like to notify users of how much the program read of the file, so I need to iterate through the whole file once and read how many lines it has. This bothers me, because it means I have to iterate a file two times.

One approach I thought about was get the byte size of first line and then divide the size of the file by that size. It would give me estimated count of lines in a file, but I'm not really sure if the approximation error wouldn't be too big.

  • A producer consumer with a throttle an the producer if it gets too far ahead. – paparazzo Jan 22 '17 at 16:26
  • I miss the loop over all chunks of data in your code. In the current form, the code will only read the first chunk. When you add the missing loop, you have to make sure the calls to _database.Insert stay in the right order, even when multiple threads are involved. Or, you have to make sure the order of insert does not matter. – Doc Brown Jan 22 '17 at 20:21
  • ... now things got even more confusing. Inserts by column? Still not showing where the code loops over the chunks? Voting to close as "unclear". – Doc Brown Jan 23 '17 at 21:21
  • ... is this a try to fix the question? Inserting all rows of one chunk still does not show how multiple chunks are processed. If you do not understand my comments, ask instead of applying wild guesses. – Doc Brown Jan 24 '17 at 6:47
  • @DocBrown I fixed what I could before going to work. I've just added the rest. I don't need exact line by line fix. I just want to know if my thinking is good in terms of keeping everything separate and clean. – FCin Jan 24 '17 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.