Methods to increase the amount of data sent in a packet

Question

I have been working on launching high-altitude balloons (HABs, or weather balloons) and I have been using LoRa to enable long-range communication with my balloons. It's been great and pretty reliable, but in the best case, I get about 50 kbps and usually a bit less or a lot less when we start getting to really far horizontal distances. To be safe I try and limit myself to 10kb-15kb per second, so about 10,000 bits worth of data per second.

I am recording this set of data, with example data:

• GPS data (67.1234, -23.9874)
• Accelerometer data (9.8066, 2.3874, -1.4235)
• Magnetometer data (-56, 39, 131)
• Temperature data (my thermometer reports in Kelvin so 262.15, or I could convert it to Celcius -11)
• Pressure data (61.456)
• Humidity data (41.27)

Now, I don't need all of that data during the flight since it all gets logged to the flight computer (a Raspberry Pi). I would say the most important (in order too) would be GPS data, accelerometer data, and pressure/temperature data (for altitude). Right now I am using a custom layout that is based on JSON, the JSON would look like this

{
   "GPS":[
      67.1234,
      -23.9874
   ],
   "Accelerometer":[
      9.8066,
      2.3874,
      -1.4235
   ],
   "temp":262.15,
   "Pressure":61.456,
   "Magnetometer":[
      -56,
      39,
      131
   ],
   "Humidity":41.27
}

Now obviously sending JSON is pretty dumb (that JSON is 1600 bits) so I minimize it down into pretty much the same thing, but without headers and I just assume the format. That string looks something like this

[67.1234, -23.9874][9.8066, 2.3874, -1.4235][262.15][61.456][-56, 39, 131][9.8066, 2.3874, -1.4235][41.27] (this is about half the size at 850 bits in text)

The problem is I have to convert the string into binary which is about 1600 bits raw. I am using my own implementation of LZW encoding which can get pretty decent results, and if I run the example string through my encoder it can get down to about 700 bits. That's not too bad and technically that would give me about 15 data transmissions per second with my current payload. I am happy with that transmission rate (I normally send data about 10/second) but if I wanted to add more sensors, or more accurate sensors, or wanted to send pictures I would very quickly run out of bandwidth.

Does anyone have any ideas to structure my payload better to get more data per transmission, better encoding methods, or any other ideas to get more data sent? I thought about just getting a better antenna but I would like to optimize my software better if possible. I would also like to add some sort of encryption if possible since I don't control all of the LoRa gateways, but that's not super important.

Answer 1

The main method to reduce your packet size is to use a binary encoding of your data.

If your sensors produce the data in a binary format, note the specifications of the format, such as the number of bits before and after the binary point. Then put the binary data as-is in the packet.

If your sensors only provide text output, use the specified range and resolution to determine a fixed-point with the least number of bits that can represent the full range and resolution and use that in your packet.

To get the absolute smallest packet, you can use bit-packing. Instead of starting each measurement on a byte boundary, you pack all measurements one after the other as a bit stream (e.g. 29 bits longitude, 29 bits latitude, 9 bits temperature, etc.)

Answer 2

With all due respect, your data is boring. So take advantage of that. Leverage its very boring nature.

I recommend you send such data in routine Update messages. And that you occasionally send Baseline messages, perhaps a few times each minute. Send each given Baseline message more than once, in case of occasional packet loss, since it's needed to properly decode other Update messages. Give each distinct Baseline message an incrementing serial number, and send the serial as part of each Update. The analagous MPEG concept is key frames or I-frames.

A GPS coordinate needs several bits to represent any location on the surface of the Earth. But you know roughly where your balloon is. So offer detailed (lat, long) information in each Baseline message, and then Updates only need to express a small delta, typically less than 1000 meters.

For that matter, you might find it convenient to have the onboard Raspberry Pi translate from degrees, and communicate in units of meters.

The raw temperature and pressure data will be sent in each Baseline message. But you explained that what you really use them for is altitude. (And apparently the notoriously bad GPS Z-coordinate has such low resolution that you don't even bother to send it.) Clearly you have some altitude(temp, press) formula that post-processing will run on the ground, to get a time series of (x, y, z) coordinates. Consider computing such altitudes on-board, include it in each Baseline, and send only that computed altitude in each Update, so temperature and pressure are elided from Updates.

More generally, your balloon's physical position corresponds to a smoothly differentiable function of time. We can wrap a Kalman filter around it. Suppose we know a recent (vx, vy, vz) velocity vector from a previous time step. It is a very good estimate for what the vector will be in the next time step. And even if it's off, it will be off by only a small delta, which can be transmitted with fewer bits.

summary

Send all the data that you're currently sending in occasional Baseline messages.

Compute altitude on-board. Compute 3-space position, and velocity vector, on-board, and perhaps model them with a Kalman filter.

Send small deltas in frequent Update messages.

---

triggered updates

An isochronous communication schedule would have boring timestamps, like {noon + .1s, noon + .2s, ..., noon + .9s}.

Sometimes your RPi interrogates a sensor, and nothing changed. (For example, humidity probably doesn't need to go in Updates, and might appear many times with same value in successive Baselines.) Sending "delta was zero", "delta was zero" consumes bandwidth while communicating very little beyond "payload is still powered up".

Each GPS reading might be distinct, but the low order bits might be very noisy, something you'd like to smooth out on the ground and perhaps on-board. This might refine your notion of whether repeated observations were "distinct".

If you query your sensors, and find that "nothing changed" since last time, consider suppressing a boring Update message. Keep polling, and re-polling, till the instant something changes -- then you send an immediate Update. Now the timestamp is actually meaningful, it lets you increase the resolution of your measurements, by narrowing down just when a sensor incremented its reported value.

Consider having two types of Update message, "fast" and "slow", which report on different sensors according to their anticipated rate of change.

modeling

Notice that you can replay your logs of previous historic balloon flights when trying to incorporate these ideas. Choose certain number of bits, or update strategies, try it against logged data to see how compression performs, and then tweak the parts you're not yet happy with.

---

simple compression

Suppose you wish to change your stack as little as possible. You can still stick with JSON + LZW. One improvement is a standard dictionary. Create a short boilerplate JSON document that mentions the names "GPS", "Accelerometer", etc., and always play that as the first part of what you send, before the real JSON payload data. It populates the LZW dictionary. It is constant, so it produces constant compressed output. Subsequent JSON text can exploit the dictionary references.

Compute compress(boiler + data1) and compress(boiler + data2). Compare the bytes and notice the common prefix. Write a layer, a pair of functions which can strip and restore that common prefix. Now the packets you send every 100 msec are shorter.

One step fancier is to replace fixed boilerplate with data values corresponding to a point in a scheduled balloon flight.

And fancier than that would be to send occasional Baseline documents reflecting "a point we visited ten seconds ago" along with Update documents which use them. So if humidity doesn't change for ten seconds, the humidity field gets squeezed right out of most Updates.

---

images

Video data versus telemetry is a whole other ball of wax, orders of magnitude more data. You already store images onboard in hopes of later recovering the balloon, and clearly that will remain your biggest source of imagery bandwidth.

Go review historic images taken on previous flights, and decide which images you would find most valuable if they were available in near realtime. I'm guessing that images early in the flight are relatively uninteresting, and images just prior to landing will be most informative, as an aid to recovering the balloon. So maybe we send no image data in the first few minutes of any flight.

Now take an image you wish you'd had in realtime, convert to grayscale, and squish its JPEG resolution. Be sure to use progressive rendering. Determine how far you can squish it while still being valuable.

Now pick a 2nd or 3rd such image. Maybe they correspond to the last two or three minutes of flight? And we'll try to send one per minute?

During flight, snap an image and chop it up into segments that take about one second to send. Alternate between sending segments and sending Update messages.

Once the balloon is on the ground, does it still have decent bandwidth to the chase car? It could continue to send "final descent" images for a few minutes while it awaits pickup.

radio bandwidth

You mentioned that you're being conservative in how you configure your radio.

Consider having low- & high-risk segments of a mission. You are nearly guaranteed to obtain some realtime data early on in the low-risk portion, even if the chase car finds the balloon at the bottom of a lake.

And then you cross your fingers, increase the transmission rate, and start sending image data during the high-risk portion. You receive those images, or you don't. Nothing ventured, nothing gained.

Maybe have the RPi reduce the transmission rate, for lower risk, when it falls below some threshold altitude. That should give you a better chance of sending GPS coords to the chase car, while you're still high enough to enjoy good radio signal propagation.

Answer 3

Many compression schemes use the fact that consecutive samples are often closely related, and the differences may use less bits than original data. If you transmit one full data packet followed by a number of incremental packets the average bits per packet may be much lower.

Answer 4

There are many binary formats for structured data. Wikipedia has a list. Some examples are EXI, CBOR, FlatBuffers, Protocol Buffers.

The are simpler to implement than a compressor, and simpler to decode, and probably compact enough for simple binary data (such as a bunch of floats or halfs).

Using a standard (container) format has the benefit of allowing for off-the-shelf code, interoperability, less risks (e.g. versioning), archivability, etc.

Update: There are even tools to describe, (de)serialize and visualize formats. Kaitai Struct is such a tool, and they have a concise comparision of a good part of the serialization "design space".

Answer 5

Your json has an advantage over binary. Things can be left out. If, for example, the Accelerometer has nothing new to say durring this sample you can leave it out.

However, if GPS, Accelerometer, temp, Pressure, Magnetometer, Accelerometer, Humidity are going to show up each and every time, each with exactly the same number of fields then text isn't doing you much good. All it's doing is letting you read it with an editor. What you have is a fixed record. And those encode into binary very nicely. You don't have to keep telling us that 3rd, 4th, and 5th numbers are from the accelerometer because they're always from the accelerometer. That lets you completely remove all the meta information about what each number is and just send the numbers. You can document what each is at your base station. Meta info doesn't need to fly.

That saves a lot of data right there. You get a little bit more compression turning text into "binary" (floats, ints, etc). Things can get lossy here though so follow Barts advice and look into the original forms this data took. Lossy isn't always bad. But you should always be aware when it's happening.

You can look into what a canned compresion algorithm can do for you. But if your record is always the same data points try this and measure it against the canned compresion algorithm. You might be suprized by how good you can do without it.

Bit packing can squeze more data in but remember to end on a byte boundry. Don't kill yourself shoving numbers together if it doesn't save you at least 8 bits a record.

Also, be aware of the law of diminishing returns. This hole is bottomless. Always check if what you're getting out of this is worth what you're putting into it. Or you'll never stop optimizing.

but if I wanted to add more sensors

Add them at the end. This is called single data inheritance. This way the position of the old numbers doesn't change and can be read by old code.

more accurate sensors

Oops, if this changes the number of needed bits you'll have to change the format and have to write new code to read it. This issue creates the need to express which format your data is in somewhere.

And that brings me back to the advantage json has over binary. Not only can things be left out, they can be added. Compressed json can be decent. It's very flexible, and maintenance friendly. But sooner or later every coder tries hand rolling a compressed format. It's good to see for yourself what makes a difference and how much difference it makes. Keep a close eye on what you're losing as you pursue this.

Answer 6

Consider zstd for compression. It has custom dictionaries, besides many other good things. Killer feature is you can train a dictionary on all those "JSON" packets collected earlier, getting a highly-tailored dictionary

Answer 7

Some interesting answers here, but as far as I can see, any of them misses the elephant in the room: you are already using an LZW compression algorithm, which is eliminating most of the redundancies in your data.

This can mitigate a lot of any of the suggested micro-optimizations of the input data size - when the input data is already a highly packed binary, LZW will not compress as much as when the input data is a not-so-highly packed JSON.

Of course, that means when you find ways to reduce your input data size by, lets say, 30%, the LZW compressed data size will still yield to a gain between 30% and 0% in output size, but I would expect it way more near to 1-5%% than to 30%.

One point which might be worth a look is your own LZW implementation. I made the personal experience that own implementations of these algorithms will rarely beat existing, popular compression libraries/tools, which have been optimized over decades. For example, did you try to use the LZMA SDK of the well-known tool 7zip? It is highly portable, has a liberal license and is one of the best general-purpose compression algorithm implementations I know.

Answer 8

Multiply each number by a fixed number of decimals to remove the decimal point. Encode each in a fixed number of binary bits big enough for every case. If you need ASCII then use base-85 encoding.

If the format changes add a 6 bit field for one of 64 data formats at the start.

If you send data 10 times per second then you can send the difference to the previous set, saving quite a few bits. Send the complete data once per second.

Answer 9

It strikes me as odd that someone who implements his own compression to minimize payload uses text to represent his raw data. But to be fair, given this pipeline, you are not likely to gain much from starting with a compact binary record.

Another thing that puzzles me is that I don't see any timestamps. Is it all realtime and does the receiver add the timestamps based on the moment of reception?

I would focus on getting faster and more reliable transmissions, software does not seem to have much to offer here.