8

We have a requirement in our project to generate a big CSV file every 2 hours using a Java program.

This file will have around 60,000 lines (around 120 characters per line). I am not sure about the size yet.

I would like to know if I would run into any memory issues because I will open the file using FileWriter and then keep writing and then finally close the file.

Should I worry about the size of the file? If yes, are there any other good techniques to write into a big file in Java other than using FileWriter?

We are using Java 5.

Improve this question
3
  • What version of Java are you using?
    – Martijn Verburg
    Commented May 8, 2012 at 13:39
  • Yo... I mean Nes. Damnit, lemme get my magic 8 ball.
    – Ripped Off
    Commented May 8, 2012 at 18:10
  • If speed matters, try a really big (multi-megabyte) buffer. It sped my file writing by a factor of 10. Of course, your results may vary....
    – RalphChapin
    Commented May 10, 2012 at 16:56

4 Answers 4

Reset to default
14

No, you shouldn't. The point of a file is to store things outside random access memory; the FileWriter's size is constant, and likely to be pretty small, all things considered, even if it's a buffered FileWriter. The constant rewriting might cause I/O load or CPU spikes, but almost certainly not memory shortage.

Improve this answer
2
  • I thought the file data will be kept in memory until I close the filewriter? looks like my assumption is wrong.
    – java_mouse
    Commented May 8, 2012 at 13:43
  • 8
    Nope, a BufferedFileWriter will keep a certain amount of data in memory, but that amount is a fixed quantity, depending on the environment it's called in. It doesn't grow unbounded with the number of bytes you push through it - that would be a recipe for disaster!
    – Kilian Foth
    Commented May 8, 2012 at 14:05
8

As Killian Foth wrote it, you should have no problem at all, 60000 lines is not at all that big. I just wanted to suggest you to use any of the free CSV parsers provided here under the "Commons CSV" initiative at http://commons.apache.org/csv/ instead of writing your own implementation.

I've used Super CSV for a few projects and I certainly did not have any problem with it.

Improve this answer
2
  • 1
    I used openCSV. I like it. 60K lines is nothing. My old laptop processes that in a second.
    – ahoffer
    Commented May 8, 2012 at 16:29
  • Great to hear Jalayn! We've just released a new version of Super CSV with heaps of bug fixes, new features and a brand new website. Oh, and it's now in Maven central :)
    – James Bassett
    Commented Nov 7, 2012 at 21:24
5

Do not use FileWriter. Not because of performance concerns (Java's IO classes do not keep everything in memory, ut 60k lines is nothing even if they did), but because it does not allow you to choose the character encoding. It will implicitly use the platform default encoding, which means text outside ASCII can get corrupted.

Instead, use an OutputStreamWriter wrapping a FileOutputStream. Or, even better, a CSV library, which should handle all these issues.

Improve this answer
3
  • Or instead of OutputStreamWriter, use NIO classes (FileChannel with ByteBuffers)? Do all the CSV libraries handle encoding? I took a quick look at SuperCSV, and didn't see anything about handling encoding.
    – Sam Goldberg
    Commented May 8, 2012 at 16:04
  • 1
    @Sam Goldberg: you're right, it seems to operate on Reader/Writer and leave that concern to the caller.
    – Michael Borgwardt
    Commented May 8, 2012 at 17:39
  • @MichaelBorgwardt You're right - Super CSV was written using IoC, so it's up to you to supply a reader/writer - that way you can write to a file, zip file, HTTP response, etc. We've just released a new version - please check it out :) Oh, and as for character encoding, I've always found Joel Spolsky's article on Unicode to be excellent.
    – James Bassett
    Commented Nov 7, 2012 at 21:18
1

You might consider using BufferedWriter, though that probably won't help significantly with performance, it's a best practice in any case, since I imagine the number of lines won't always be 60,000.

Have you considered zipping the file afterwards? If you intend on having a lot of these files laying around, might be in your best interest to zip it up after it's been written to, especially if you're going to be creating these files once every couple hours.

For what concerns memory, you probably have nothing to worry about unless you're working on a system with very little memory, in which case you should use BufferedWriter and explicitly set the buffer size.

Improve this answer
2
  • 1
    What's a BufferedFileWriter?
    – Michael Borgwardt
    Commented May 8, 2012 at 14:34
  • Oops. I meant BufferedWriter. Fixed.
    – Neil
    Commented May 8, 2012 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.