Skip to main content
33 votes

Methods to increase the amount of data sent in a packet

The main method to reduce your packet size is to use a binary encoding of your data. If your sensors produce the data in a binary format, note the specifications of the format, such as the number of ...
Bart van Ingen Schenau's user avatar
19 votes
Accepted

What is the encoding used in Git's binary patches?

Aha, it's RFC1924's version of the base85 encoding, which uses 5 ASCII characters to represent 4 bytes (80% efficiency): static const char en85[] = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '...
Dan Lenski's user avatar
18 votes

Methods to increase the amount of data sent in a packet

With all due respect, your data is boring. So take advantage of that. Leverage its very boring nature. I recommend you send such data in routine Update messages. And that you occasionally send ...
J_H's user avatar
  • 7,320
10 votes
Accepted

UTF-8 questions

Answer to Question 1: why is it that the first 2 bits of every continuation byte are set to “10”? It lets you land at a random place in the sequence and unambiguously work back to the beginning of ...
SusanW's user avatar
  • 1,035
8 votes

Methods to increase the amount of data sent in a packet

Many compression schemes use the fact that consecutive samples are often closely related, and the differences may use less bits than original data. If you transmit one full data packet followed by a ...
Hans-Martin Mosner's user avatar
5 votes

Methods to increase the amount of data sent in a packet

There are many binary formats for structured data. Wikipedia has a list. Some examples are EXI, CBOR, FlatBuffers, Protocol Buffers. The are simpler to implement than a compressor, and simpler to ...
Pablo H's user avatar
  • 655
4 votes

Methods to increase the amount of data sent in a packet

Your json has an advantage over binary. Things can be left out. If, for example, the Accelerometer has nothing new to say durring this sample you can leave it out. However, if GPS, Accelerometer, temp,...
candied_orange's user avatar
3 votes

Methods to increase the amount of data sent in a packet

Consider zstd for compression. It has custom dictionaries, besides many other good things. Killer feature is you can train a dictionary on all those "JSON" packets collected earlier, getting ...
Maksym Brashchenko's user avatar
3 votes

Compressing EBCDIC file vs UTF8

First thoughts EBCDIC is a format with an 8 bit encoding. The basic EBCDIC character set has plenty of unused space, so not all the potential of the 8 bits is used. But it has codepages for handling ...
Christophe's user avatar
  • 78.4k
3 votes

Why does UTF-8 waste several bits in its encoding

What hasn't been mentioned is that if you have a correct sequence of code points, and a pointer that is guaranteed to point to the first byte of a code point, with UTF-8 you can very easily find the ...
gnasher729's user avatar
  • 45.8k
3 votes

Reduce number of digits by converting to alphanumeric data

This is precisely why storing business defined identifiers as database identifiers becomes problematic. The business changes their rules, and this forces downstream changes in databases. If you were ...
Greg Burghardt's user avatar
3 votes
Accepted

Name of data structure where a string determines its value

Type-Length-Value or Tag-Length-Value. Both protocols describe a mechanism for encoding optional information in multiple tuplets of variable length into text-based messages, using three fields: type, ...
Robert Harvey's user avatar
2 votes
Accepted

Fast and simple hex compression

The current protocol encodes the data into hex What protocol? TCP certainly doesn't do that. I believe you're actually referring to binary to text encoding. This is a method of converting binary ...
candied_orange's user avatar
2 votes
Accepted

How to identify encoding of a text string?

@MartinGrey, I think your question is perfectly sensible, but it's like Bob says in a comment, your misconception is in the idea that handling arbitrary data in arbitrary encodings, and decoding it ...
Steve's user avatar
  • 9,012
2 votes

Reduce number of digits by converting to alphanumeric data

Encode the number as base 64 using a fixed size representation and restrict one of the characters to be non-numeric to prevent collisions with existing identifiers. Base 64 uses 'A'-'Z' to represent 0-...
ggf31416's user avatar
  • 151
1 vote

Methods to increase the amount of data sent in a packet

Multiply each number by a fixed number of decimals to remove the decimal point. Encode each in a fixed number of binary bits big enough for every case. If you need ASCII then use base-85 encoding. If ...
gnasher729's user avatar
  • 45.8k
1 vote

Methods to increase the amount of data sent in a packet

Some interesting answers here, but as far as I can see, any of them misses the elephant in the room: you are already using an LZW compression algorithm, which is eliminating most of the redundancies ...
Doc Brown's user avatar
  • 209k
1 vote

Reduce number of digits by converting to alphanumeric data

Unclear elements It would be useful to clarify the following points: Is the - currently stored in the string ? Is the length control on the client side based on the visible string or on the ...
Christophe's user avatar
  • 78.4k
1 vote
Accepted

Reduce number of digits by converting to alphanumeric data

When this really solves the problem, why not? Sometimes, a pragmatic solution with some "duct tape" is required to get things done, even if it does not look beautiful from a design perspective. ...
Doc Brown's user avatar
  • 209k
1 vote
Accepted

Dictionary Coding for OLAP

Let us start with your "products" as a representative example: OLAP databases which involve products are typically used to create queries like which are the top 10 selling products over the last 3 ...
Doc Brown's user avatar
  • 209k
1 vote

Windows compatibility with Unix/Linux newline "\n"

This question is really about a software application's "customer base". To answer your question, you have to know whether your customers might be inconvenienced if your application generates output ...
rwong's user avatar
  • 16.9k
1 vote

Windows compatibility with Unix/Linux newline "\n"

As far as Windows and C# is involved you can always use the Environment.Newline to determine the default new line character of the system the program is ran on. also, you can use text.Replace("\n",...
Xeorge Xeorge's user avatar
1 vote

Windows compatibility with Unix/Linux newline "\n"

Windows Notepad (notepad.exe) doesn't interpret a standalone \n as a new line. It's not necessarily "modern" but pretty much "mainstream". If you're writing text files, every day user should be able ...
Mario's user avatar
  • 1,509
1 vote

How is encoding handled correctly during copy-paste between programs?

A program transferring text can do one of three things: Transfer without telling which encoding is used, transfer using a standard encoding, or transfer while specifying the encoding. The receiving ...
gnasher729's user avatar
  • 45.8k

Only top scored, non community-wiki answers of a minimum length are eligible