3

I have been working with some fairly large text files containing about two million lines of text. I don't know the length of the content or the lines in advance, just the number of lines. I have been considering two problems that probably have a similar solution:

  • How do I change or add to the nth line, without my computer having to shift everything after the nth line in the computer memory.
  • Say I want to delete all the lines in the text file except for the last 100k, every time the number of lines grows beyond two million. How do I keep these last 100k lines, without having to read the entire text file, deleting everything, and rewriting the last 100k?

I suppose it could be solved by putting everything in some database system. However, as that is no option for me, I want to know how to work with text files efficiently.

marked as duplicate by Kilian Foth, Community Oct 14 '15 at 11:45

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

4

How do I change or add to the nth line

You can't.

You have lines of different lengths so you cannot overwrite a chunk of the file, "knowing" it to contain an entire "line".

If your records were all the same length, then you would probably be able tom using direct file access.

How do I keep these last 100k lines, without having to read the entire text file, deleting everything, and rewriting the last 100k?

You can't.

You don't know where each line begins and ends, so you have to read them line-by-line, starting at the beginning. In this case, the best you can do is to create a new file containing only the last 100k lines and then overwrite the original file with it.

Looking beyond your immediate problem(s), you're now well into the sort of sizes where a database would massively simplify your code and massively improve your application's performance.

0

Load entire file into memory. You must represents all lines as a linked list. First line links to 2nd line. 2nd line links to 3rg line etc. If you delete line 2, you just change link in first line from line 2 to line 3. You can edit each line w/o move 100k lines each character instert etc.

Not the answer you're looking for? Browse other questions tagged or ask your own question.