From what I remember, the JPEG and MPEG/H264 family of motion codecs internally represent the images in YCbCr colorspace (not RGB), where Y is the luminance, and Cb with Cr are the chrominance components. While every pixel of Y component is used, the chrominance is downsampled according to one of a few possible schemes. The main compression gain in JPEG (and keyframes of motion codecs) is achieved by quantization of DCT results, which means throwing away the high frequency components of each image block.
To compress the differences between frames caused by motion, the motion prediction is used in the luminance component (before DCT). Very briefly, each frame is divided into regular blocks of pixels, and each block of current frame is compared with pixels of the previous frame to find the closest match. This produces the deplacement vector (motion vector). Because the match is not ideal, the pixel value differences in each of the Y, Cb, and Cr components are passed through the normal JPEG compression (DCT, quantization, Huffman encoding) and sent along the motion vector to form a Delta Frame.
Here, you will find a few hints for your further research. I hope you'll not have to reinvent the wheel. There are some Linux/Unix image processing tools you might harness for your task.