I have a folder containing about 9,000 JPEG photos (about 30Gb), which I want to archive with some sort of compression. I understand that compressing JPEGs is not normally very effective, but these photos are frames of a timelapse, so there is a huge amount of commonality between most of the images. Am I likely to get any more filesize reduction than normal in this case? Is there a particular (common) compression algorithm that's likely to do particularly well in this scenario?
That's an interesting question: can popular compression algorithms still make use of the redundancy in frames after they've been individually compressed, or is the individual compression too good to "leave traces"? I don't know, and you'd have to try it out to get a reliable answer.
However, it's almost certainly a better idea to store all these frames as a video stream in the first place, because video encoders are written specifically for the job. (Reusing successful libraries is almost always faster and more efficient than rolling your own.) You can always extract each individual frame from the video if you need it. I don't think you can find a standard compressor that will do nearly as well at this task as the ones that are custom-tailored for the job.
Methods of data compression that exploit redundancy between individual data groups of a set (usually a set of similar images) are named Set Redundancy Compression (SRC was proposed firstly by Kosmas Karadimitriou in 1996).
There are four well-known types of SRC techniques:
- Min-Max differential method (MMD)
- Min-Max predictive method (MMP)
- centroid method
- multilevel centroid method
A Comparison of Set Redundancy Compression Techniques (Samy Ait-Aoudia and Abdelhalim Gabis) contains a brief description and comparison of the various algorithms.
MMP methods usually perform better than the other SRC techniques.
SRC is an active field of research but you'll hardly find a ready to use software.
Since your photos are frames of a timelapse, you have high "temporal redundancy" and video compression methods can be used effectively. However:
- an image can not be decompressed without decompressing a number of other images;
- most of the video compression methods are lossy.
Also consider that JPEG files can be further compressed: software based on context mixing algorithm (e.g. PAQ lossless data compression archivers) are quite good in this regard (at the expense of speed and memory).