My requirement is to store a large number of files within a single file.The files stored could be anything like images, videos or simple text files as well. I want some ideas to implement the same. I am thinking of implementing a file system within a file, but am not sure if its a good idea.

Adding in more details as requested : Platform to be developed on is Android. The idea initially was to store all the data using sqlite and provide encryption on it, but I think it will eventually lead to a slow down as the file size increases. The file is going to increase in size with time.

The main area of concern here is the access time. Also I want to provide encryption for this single file. Any suggestions are welcome.

  • Do you need to store text files? – superM Dec 5 '12 at 18:29
  • 2
    The ZIP format (en.wikipedia.org/wiki/Zip_%28file_format%29) can do that, although it also compresses the files so I'm not sure it will be very good for access time (unless you use minimal compression). Maybe TAR files would be more appropriate? en.wikipedia.org/wiki/Tar_%28computing%29 – FrustratedWithFormsDesigner Dec 5 '12 at 18:29
  • @FrustratedWithFormsDesigner: please make your comment an answer, I will upvote it :) – 9000 Dec 5 '12 at 18:41
  • 1
    @9000: Thanks but I'm not actually sure if these will be good formats based on the "access time" requirement. I was suggesting the OP to read up on them for further research. If you know that these formats are good for OP's purpose (and can back that up), then I think you should post the answer and I'll upvote. ;) – FrustratedWithFormsDesigner Dec 5 '12 at 18:43
  • 1
    See also stackoverflow.com/questions/2933581/… – user40980 Dec 5 '12 at 19:11

Note: if you stated the purpose of your file container more clearly, describing access patterns, desired platforms, and the problem you're solving in general, the answers might be better.

Your description looks awfully similar to a game resource file, a renowned file type. These files are not intended to be updated frequently (if ever), but are optimized for fast seeking and reading.

There are several known implementations: for instance, iD Software used WAD files and PAK files, but finally came to use ZIP files.

Many applications use various derivatives of IFF format, built from self-describing chunks, and, with some care, efficiently updatable.

In a chunked file, or a zip file (consider zero compression), you can encrypt each entry independently, before writing it into the file. Provided that you use a block cipher like AES256, your data does not change size, except for aligning it to block boundary. You definitely want your decryption key stored somewhere else :)

Writing a file system to provide encryption is not only possible, but has actually been done many times. For instance, Linux has encfs, and Windows has encrypted folders. TrueCrypt is an advanced virtual encrypted FS available on many platforms.

Please note that while zip files provide for a sort of native encryption, this encryption is relatively weak. Same applies to rar files, for all I know.

| improve this answer | |

Google confirms it is good idea in some cases: https://www.google.ru/images/nav_logo114.png

In this example Google optimises with reducing http requests number. User downloads one image file containing few navigation sprites and then browser splits and places it with CSS and JS.

| improve this answer | |

but am not sure if its a good idea.

It seems to be! You don't say what operating system you're working with. In Unix-based operating systems, a directory already is a kind of file. So, two things to consider:

  • You may be able to just mark this requirement 'done' and move on.

  • If you decide to go ahead with the project, you'll need to think carefully about how your implementation can be faster than existing ones.

| improve this answer | |

Have you considered jar format? It's java-specific but then it would easily port to multiple operating systems. Drawback is your users would need to have java on their machines to do anything with the files after downloading and you would probably have to do work to make that automatic/transparent to them. You would also have to handle the encryption/decryption in some way.

| improve this answer | |
  • 3
    Jar is just a renamed zip. – 9000 Dec 5 '12 at 19:37
  • @9000 - fair enough, but do other options allow zips without compression? – Matt Dec 5 '12 at 19:41
  • 3
    @Matt zip -0 or zip --compression-method store (these are synonymous of each other). – user40980 Dec 5 '12 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.