Timeline for How to simulate inner join on very large files in java (without running out of memory)
Current License: CC BY-SA 3.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 5, 2014 at 2:34 | answer | added | InformedA | timeline score: 0 | |
| Sep 24, 2013 at 16:53 | comment | added | Sean McSomething | @Constantin - "The overhead of these initial inserts and subsequently writing the results out back to files might be costly in terms of performance" - so is sorting large text files on an arbitrary key. | |
| Sep 24, 2013 at 13:03 | history | edited | Constantin | CC BY-SA 3.0 |
added 7302 characters in body
|
| Sep 6, 2013 at 19:22 | vote | accept | Constantin | ||
| Sep 6, 2013 at 19:21 | history | edited | Constantin | CC BY-SA 3.0 |
added 1930 characters in body
|
| Sep 6, 2013 at 19:10 | comment | added | Constantin | Clockwork-Muse - this particular program supports the processing of files, not database tables. What you suggest would require that I bulk insert both files into an embedded database before executing the join. The overhead of these initial inserts and subsequently writing the results out back to files might be costly in terms of performance | |
| Sep 6, 2013 at 14:39 | comment | added | Clockwork-Muse | Err... why? In any case, I'd recommend looking at the source for one or more of the in-memory RDBMSs written in Java, which thus actually do it. | |
| Sep 6, 2013 at 14:26 | answer | added | ratchet freak | timeline score: 3 | |
| Sep 6, 2013 at 14:08 | comment | added | Constantin | let us continue this discussion in chat | |
| Sep 6, 2013 at 14:02 | comment | added | Constantin | yes, as mentioned previously, I have sorted these using an external sort routine | |
| Sep 6, 2013 at 14:01 | comment | added | ratchet freak | are the files sorted on the key? | |
| Sep 6, 2013 at 13:59 | comment | added | Constantin | OK, so I will need to convert the right file to be an instance of RandomAccessFile, and when i do the inner join part, I keep a file pointer to the current row (using getFilePointer) and with each iteration, call seek(pointer) to return to that position. Is this the idea and is this the most efficient way if so? | |
| Sep 6, 2013 at 13:40 | comment | added | ratchet freak | you can do a readLine and it will work fine ;) | |
| Sep 6, 2013 at 13:38 | comment | added | Constantin | My understanding is this is used for binary files, not lines of text (in my case tab delimited with newlines at the end of each row) | |
| Sep 6, 2013 at 13:29 | comment | added | ratchet freak | RandomAccessFile allows you to remember the some locations and return to it | |
| Sep 6, 2013 at 13:24 | history | asked | Constantin | CC BY-SA 3.0 |