Timeline for How do I efficiently read random lines from a TXT or CSV file?
Current License: CC BY-SA 4.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 21, 2021 at 3:00 | history | tweeted | twitter.com/StackSoftEng/status/1352088709068369920 | ||
| Jan 17, 2021 at 16:59 | history | edited | jaylawl | CC BY-SA 4.0 |
final note added
|
| Jan 17, 2021 at 10:20 | history | edited | jaylawl | CC BY-SA 4.0 |
added clarification for solution
|
| Jan 17, 2021 at 10:15 | vote | accept | jaylawl | ||
| Jan 17, 2021 at 8:49 | answer | added | Arseni Mourzenko | timeline score: 2 | |
| Jan 17, 2021 at 8:35 | history | edited | jaylawl | CC BY-SA 4.0 |
even more context
|
| Jan 17, 2021 at 8:27 | comment | added | jaylawl | @ErikEidt I have attached some additional context to my post; hope it helps! | |
| Jan 17, 2021 at 8:26 | history | edited | jaylawl | CC BY-SA 4.0 |
fix typo
|
| Jan 17, 2021 at 1:54 | comment | added | Erik Eidt | Your problem is underspecified. What do you mean be efficient? More importantly, what is the access pattern: just one random entry here and there from random sources, or many, many in a series, based off of unchanging data? You refer to grabbing a single random puzzle, but in context is that really the case? Do you need to know the case number of the random test? Is the case number encoded in the text file, or can that only be found by knowing the ordered record number? Beyond these, why is 700MB an unworkably large amount of RAM these days? | |
| Jan 16, 2021 at 20:36 | review | Close votes | |||
| Jan 21, 2021 at 3:09 | |||||
| Jan 16, 2021 at 20:35 | history | edited | Robert Harvey | CC BY-SA 4.0 |
deleted 23 characters in body; edited title
|
| Jan 16, 2021 at 20:19 | answer | added | Christophe | timeline score: 5 | |
| Jan 16, 2021 at 17:38 | comment | added | Kilian Foth | Indexing. (One-word comments are not allowed, but really the only word you need is "indexing".) | |
| Jan 16, 2021 at 16:31 | comment | added | amon | Scan the database once to create an index of entries, in particular noting the byte offsets where the entries start. The index will fit into memory. Scanning can be performed in chunks and doesn't need to fit the entire database into memory. The index is sufficient to find the offset of an entry, you can then seek() to the correct position in the database file. You can have all of this for free by using an actual database engine, e.g. embedding HSQLDB in Java. | |
| Jan 16, 2021 at 16:24 | comment | added | Erik Eidt | Try opening the file and reading from a random location until you get a complete line. | |
| Jan 16, 2021 at 16:20 | history | asked | jaylawl | CC BY-SA 4.0 |