In my application, I'm getting product information files from an FTP server. They are text files that have been ZIP-compressed.

At first, I thought uncompressed the file on the server then reading the text over the web would work, but the uncompressed files are ~900 MB, making it slow and inefficient while causing out-of-memory exceptions.

How should I handle this type of problem? Should I download the compressed files, uncompress and process them locally, then re-upload them? Or is there another, more efficient way to do this?

  • Please clarify the data flow in your question. Its not clear why you would need to upload it again. Is there a reason a database isnt in use? – GrandmasterB Dec 8 '11 at 5:12
  • @GrandmasterB :My boss feels reading text file from local system and saving text to server database will be too slow..and my website ftp and the ftp from where i get product information files are in different servers. – karthi Dec 8 '11 at 5:16
  • 1
    I'd love to help, but you are not describing the data flow you are looking for. The only thing I can suggest is try a different decompression library to prevent memory errors. – GrandmasterB Dec 8 '11 at 5:20
  • @GrandmasterB the data flow is zipped text files from ftp--->unzip it-->upload to website server--->read and save data in sql database...is there any better way of saving zipped text files from a ftp to sql server – karthi Dec 8 '11 at 5:26
  • 1
    How many text files does the 900mb zip archive contain? You can unzip it locally, then zip the individual text files (each should be reasonably sized) then upload them – rwong Dec 8 '11 at 5:46

I assume you want to do so programmatically.

Yo don't need to uncompress first. E.g., in Java, you can read files in a zip archive directly, using java.util.zip.ZipFile:

ZipFile zipfile = new ZipFile(filename);
Enumeration e = zipfile.entries();
while(e.hasMoreElements()) {
    entry = (ZipEntry) e.nextElement();
    is = zipfile.getInputStream(entry));
    // now read from is and write to ftp outputstream 

By doing so you avoid the overhead of uncompressing, which may save you from out-of-memory error.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.