Fast compression in C++ and decompression in C#

Question

Overview

I am working on a client-server application. The client is written in C++ (working on Windows, planning to support Linux) and the server is a .NET RESTful service. I need to HTTP POST some data to the server. The data is a unicode string that I need to compress using any fast compression algorithm (needs to be light on CPU). On the server I need to decompress raw bytes and get a string.

Problem

I cannot decompress raw bytes and I end up with decompressor's output buffers stay intact.

I tried

I have tried using Google Snappy, there is only a C/C++ version and any .NET ports don't appear to be finished. I also checked LZ4, where compression/decompression works in a single language, but when I try to use both - I cannot decompress data correctly (the decompressed bytes are set to 0s).

Question

Has anyone tried using a fast C/C++ compression and C# decompression? Any recommendations?

_{Client application shall not use more than 5% of CPU - that's a vague requirement I have. I plan to try gzip and maybe deflate, but I am not sure if I can configure compression levels there or whether it will work in both C/C++ and C#. Client collects some near real time data into a small buffer, when that buffer gets full it shall be compressed and POSTed. The whole OS process shall take less than 5% of the machine's CPU. A buffer is roughly posted 2 times a second which creates a traffic of about 2Kb/sec.}

Answer 1

If anybody interested I ended up using gzip from zlib. Never figured out why LZ4 doesn't work, as suggested in the comments this could be an endianess problem or a 64/32-bit mismatch. However, I tested this on a single machine compressing and decompressing a local file. The same compilation settings worked for gzip.

C/C++ sample compressor code

int compress_one_file(char *infilename, char *outfilename)
 {
    FILE *infile = fopen(infilename, "rb");
    gzFile outfile = gzopen(outfilename, "wb");
    if (!infile || !outfile) return -1;

    char inbuffer[128];
    int num_read = 0;
    unsigned long total_read = 0, total_wrote = 0;
    while ((num_read = fread(inbuffer, 1, sizeof(inbuffer), infile)) > 0) {
       total_read += num_read;
       gzwrite(outfile, inbuffer, num_read);
    }
    fclose(infile);
    gzclose(outfile);
 }

C# sample decompressor code

public static void Decompress(FileInfo fileToDecompress)
{
    using (FileStream originalFileStream = fileToDecompress.OpenRead())
    {
        string currentFileName = fileToDecompress.FullName;
        string newFileName = currentFileName + ".decompressed";

        using (FileStream decompressedFileStream = File.Create(newFileName))
        {
            using (GZipStream decompressionStream = 
                new GZipStream(originalFileStream, CompressionMode.Decompress))
            {
                decompressionStream.CopyTo(decompressedFileStream);
            }
        }
    }
}

Answer 2

Both LZ4 and Snappy has been ported to .NET recently.

You can find LZ4 .NET port here at http://lz4net.codeplex.com.
You can find Snappy .NET port (actually P/Invoke wrapper) here at http://snappy4net.codeplex.com.

You can also check performance comparison (of .NET ports) here.

To answer your question I did use both to test C++ compression and C# decompression and it works fine.

Disclaimer: yes, both LZ4 and Snappy has been ported/wrapped by me.

Answer 3

Take a look at 7-zip's LZMA format. here

I have only used their compression utilities (but not their SDK) but they are excellent. The compression format is portable across language and platform and they claim support for both C# and C++

Answer 4

You must develop a program that will run in the console. Your program will be called "TP2.exe" and has three parameters:

1. the name of an input file,
2. a boolean (true or false) indicating whether you want to compress the input file or unpack.
3. the name of the output file (the result will be written to this file.)

For example, if you run the program with the following command in the console: TP2.exe test.txt true test2.txt Your program will read the file test.txt as input and writes the compressed file test2.txt output. Another example: if you run the program with the following command in the console:

TP2.exe test2.txt false test.txt

Your program will read the file and write test2.txt the decompressed file test.txt output. Your program should use the method of compression / decompression simply discussed in the course is based on the use of a self-organized list.

An example file uncompressed and compressed file example is provided as an example of CLIC. For the work is simple to make, you can assume that the sentences are already cut into words and that there is a single word per line.

Finally, at the implementation level, it is recommended to use a linked list to list the self-organized because it would be more efficient. A good idea would be to use a linked list implementation of the STL library.