5

I have been working on launching high-altitude balloons (HABs, or weather balloons) and I have been using LoRa to enable long-range communication with my balloons. It's been great and pretty reliable, but in the best case, I get about 50 kbps and usually a bit less or a lot less when we start getting to really far horizontal distances. To be safe I try and limit myself to 10kb-15kb per second, so about 10,000 bits worth of data per second.

I am recording this set of data, with example data:

  • GPS data (67.1234, -23.9874)
  • Accelerometer data (9.8066, 2.3874, -1.4235)
  • Magnetometer data (-56, 39, 131)
  • Temperature data (my thermometer reports in Kelvin so 262.15, or I could convert it to Celcius -11)
  • Pressure data (61.456)
  • Humidity data (41.27)

Now, I don't need all of that data during the flight since it all gets logged to the flight computer (a Raspberry Pi). I would say the most important (in order too) would be GPS data, accelerometer data, and pressure/temperature data (for altitude). Right now I am using a custom layout that is based on JSON, the JSON would look like this

{
   "GPS":[
      67.1234,
      -23.9874
   ],
   "Accelerometer":[
      9.8066,
      2.3874,
      -1.4235
   ],
   "temp":262.15,
   "Pressure":61.456,
   "Magnetometer":[
      -56,
      39,
      131
   ],
   "Humidity":41.27
}

Now obviously sending JSON is pretty dumb (that JSON is 1600 bits) so I minimize it down into pretty much the same thing, but without headers and I just assume the format. That string looks something like this

[67.1234, -23.9874][9.8066, 2.3874, -1.4235][262.15][61.456][-56, 39, 131][9.8066, 2.3874, -1.4235][41.27] (this is about half the size at 850 bits in text)

The problem is I have to convert the string into binary which is about 1600 bits raw. I am using my own implementation of LZW encoding which can get pretty decent results, and if I run the example string through my encoder it can get down to about 700 bits. That's not too bad and technically that would give me about 15 data transmissions per second with my current payload. I am happy with that transmission rate (I normally send data about 10/second) but if I wanted to add more sensors, or more accurate sensors, or wanted to send pictures I would very quickly run out of bandwidth.

Does anyone have any ideas to structure my payload better to get more data per transmission, better encoding methods, or any other ideas to get more data sent? I thought about just getting a better antenna but I would like to optimize my software better if possible. I would also like to add some sort of encryption if possible since I don't control all of the LoRa gateways, but that's not super important.

3
  • 23
    "The problem is I have to convert the string into binary which is about 7600 bits raw" - sorry, I'm a bit tired so I may be missing something, but what do you mean by this? How did you go from 850 bits to 7600 bits??? BTW, as some other answers have suggested, if you ditch the text representation completely and send your 14 data points as a sequence of, say, 4 byte floats and ints, that's 14*4*8 = 448 bits raw data. Dec 29, 2023 at 22:43
  • 7
    Your text example is 106 characters, which is 106 bytes using ASCII or UTF-8 encoding. No idea where 7600 is coming from.
    – OrangeDog
    Dec 30, 2023 at 21:44
  • @OrangeDog sorry I meant the 1600 that I mentioned earlier in the question, but to get that I just grabbed the raw JSON (including new lines and spaces) and put it into some online calculator. I'll update the question, thank you! Jan 2 at 18:43

9 Answers 9

33

The main method to reduce your packet size is to use a binary encoding of your data.

If your sensors produce the data in a binary format, note the specifications of the format, such as the number of bits before and after the binary point. Then put the binary data as-is in the packet.

If your sensors only provide text output, use the specified range and resolution to determine a fixed-point with the least number of bits that can represent the full range and resolution and use that in your packet.

To get the absolute smallest packet, you can use bit-packing. Instead of starting each measurement on a byte boundary, you pack all measurements one after the other as a bit stream (e.g. 29 bits longitude, 29 bits latitude, 9 bits temperature, etc.)

8
  • 3
    As this post said. I might recommend ProtoBuff, which is optimized for small message sizes using binary.
    – Euphoric
    Dec 29, 2023 at 17:31
  • 3
    @Lv_InSaNe_vL I'd have to look it up, but I think protobuf's overhead in terms of bandwidth used or packet size is pretty minimal. As for the body, it's not that it has compression, but rather that it uses a binary encoding which is inherently more compact than text. The exact size depends on the data and how you define your schema, but I calculate that your 850 bits example would encode as a 336 bits protobuf message body, and that's with a straightforward schema definition with barely any attempt to optimize it for minimizing message size.
    – Douglas
    Dec 30, 2023 at 2:26
  • 6
    @Lv_InSaNe_vL protobuf doesn't send schema on the wire, it's assumed that the sender and receiver have already agreed to it. Its self-description is limited to encoding which fields are present, which is quite cheap (and valuable, because it means that you can make better use of your bandwidth by having some information in every packet, while less-important or less-quickly-changing sensors only report every Nth packet). If you add a decent entropy coder to your LZW the overhead can be less than one byte per field and you still come out well ahead.
    – hobbs
    Dec 30, 2023 at 3:52
  • 1
    Another way to think about it is that if you get down to, say, a 350 bit message with protobuf and your communications channel is good for at least 10,000 bits/sec, that's a floor of an update every 35 miliseconds. You could optimize further by creating your own binary serialization format and implementing all the data-saving techniques suggested here, but do you need to? Dec 31, 2023 at 6:52
  • 1
    Agreed, binary encoding seems like the way to go. Without knowing the specifics of how your data looks, and just looking at your example, it seems like you have 11 floating-point numbers? So if you use regular 32-bit floats for those, that's 11*4 = 44 bytes total. So 352 bits. And you can probably lose the compression completely. New sensor data or whatever would go on the end. You could shrink it more by using some short ints or medium ints or whatever, but it's probably easier to just keep it simple and say 'floats forever'. If you wanted to 'skip' a sensor, you could send a NaN for that one
    – Uberbrady
    Dec 31, 2023 at 18:31
18

With all due respect, your data is boring. So take advantage of that. Leverage its very boring nature.

I recommend you send such data in routine Update messages. And that you occasionally send Baseline messages, perhaps a few times each minute. Send each given Baseline message more than once, in case of occasional packet loss, since it's needed to properly decode other Update messages. Give each distinct Baseline message an incrementing serial number, and send the serial as part of each Update. The analagous MPEG concept is key frames or I-frames.

A GPS coordinate needs several bits to represent any location on the surface of the Earth. But you know roughly where your balloon is. So offer detailed (lat, long) information in each Baseline message, and then Updates only need to express a small delta, typically less than 1000 meters.

For that matter, you might find it convenient to have the onboard Raspberry Pi translate from degrees, and communicate in units of meters.

The raw temperature and pressure data will be sent in each Baseline message. But you explained that what you really use them for is altitude. (And apparently the notoriously bad GPS Z-coordinate has such low resolution that you don't even bother to send it.) Clearly you have some altitude(temp, press) formula that post-processing will run on the ground, to get a time series of (x, y, z) coordinates. Consider computing such altitudes on-board, include it in each Baseline, and send only that computed altitude in each Update, so temperature and pressure are elided from Updates.

More generally, your balloon's physical position corresponds to a smoothly differentiable function of time. We can wrap a Kalman filter around it. Suppose we know a recent (vx, vy, vz) velocity vector from a previous time step. It is a very good estimate for what the vector will be in the next time step. And even if it's off, it will be off by only a small delta, which can be transmitted with fewer bits.

summary

Send all the data that you're currently sending in occasional Baseline messages.

Compute altitude on-board. Compute 3-space position, and velocity vector, on-board, and perhaps model them with a Kalman filter.

Send small deltas in frequent Update messages.


triggered updates

An isochronous communication schedule would have boring timestamps, like {noon + .1s, noon + .2s, ..., noon + .9s}.

Sometimes your RPi interrogates a sensor, and nothing changed. (For example, humidity probably doesn't need to go in Updates, and might appear many times with same value in successive Baselines.) Sending "delta was zero", "delta was zero" consumes bandwidth while communicating very little beyond "payload is still powered up".

Each GPS reading might be distinct, but the low order bits might be very noisy, something you'd like to smooth out on the ground and perhaps on-board. This might refine your notion of whether repeated observations were "distinct".

If you query your sensors, and find that "nothing changed" since last time, consider suppressing a boring Update message. Keep polling, and re-polling, till the instant something changes -- then you send an immediate Update. Now the timestamp is actually meaningful, it lets you increase the resolution of your measurements, by narrowing down just when a sensor incremented its reported value.

Consider having two types of Update message, "fast" and "slow", which report on different sensors according to their anticipated rate of change.

modeling

Notice that you can replay your logs of previous historic balloon flights when trying to incorporate these ideas. Choose certain number of bits, or update strategies, try it against logged data to see how compression performs, and then tweak the parts you're not yet happy with.


simple compression

Suppose you wish to change your stack as little as possible. You can still stick with JSON + LZW. One improvement is a standard dictionary. Create a short boilerplate JSON document that mentions the names "GPS", "Accelerometer", etc., and always play that as the first part of what you send, before the real JSON payload data. It populates the LZW dictionary. It is constant, so it produces constant compressed output. Subsequent JSON text can exploit the dictionary references.

Compute compress(boiler + data1) and compress(boiler + data2). Compare the bytes and notice the common prefix. Write a layer, a pair of functions which can strip and restore that common prefix. Now the packets you send every 100 msec are shorter.

One step fancier is to replace fixed boilerplate with data values corresponding to a point in a scheduled balloon flight.

And fancier than that would be to send occasional Baseline documents reflecting "a point we visited ten seconds ago" along with Update documents which use them. So if humidity doesn't change for ten seconds, the humidity field gets squeezed right out of most Updates.


images

Video data versus telemetry is a whole other ball of wax, orders of magnitude more data. You already store images onboard in hopes of later recovering the balloon, and clearly that will remain your biggest source of imagery bandwidth.

Go review historic images taken on previous flights, and decide which images you would find most valuable if they were available in near realtime. I'm guessing that images early in the flight are relatively uninteresting, and images just prior to landing will be most informative, as an aid to recovering the balloon. So maybe we send no image data in the first few minutes of any flight.

Now take an image you wish you'd had in realtime, convert to grayscale, and squish its JPEG resolution. Be sure to use progressive rendering. Determine how far you can squish it while still being valuable.

Now pick a 2nd or 3rd such image. Maybe they correspond to the last two or three minutes of flight? And we'll try to send one per minute?

During flight, snap an image and chop it up into segments that take about one second to send. Alternate between sending segments and sending Update messages.

Once the balloon is on the ground, does it still have decent bandwidth to the chase car? It could continue to send "final descent" images for a few minutes while it awaits pickup.

radio bandwidth

You mentioned that you're being conservative in how you configure your radio.

Consider having low- & high-risk segments of a mission. You are nearly guaranteed to obtain some realtime data early on in the low-risk portion, even if the chase car finds the balloon at the bottom of a lake.

And then you cross your fingers, increase the transmission rate, and start sending image data during the high-risk portion. You receive those images, or you don't. Nothing ventured, nothing gained.

Maybe have the RPi reduce the transmission rate, for lower risk, when it falls below some threshold altitude. That should give you a better chance of sending GPS coords to the chase car, while you're still high enough to enjoy good radio signal propagation.

3
  • Hmm, I actually like the idea of just sending the altitude data (and yeah, I didn't even consider the z-axis on the GPS when it was multiple feet off just sitting on my desk haha) instead of each component. When I wrote it originally I thought it would be nice to have the data on hand but during real missions, I never look at those values. I also really like the idea of only sending GPS data periodically (maybe every second or so?) and then using acceleration data to estimate the location. Since that number isn't critical until the payload is back on the ground. Dec 29, 2023 at 20:21
  • I was re-reading your comment and I noticed the part about populating the LZW dictionary with the JSON data. But I am not actually sending the raw JSON data through LZW, I strip out all the extra info so it looks more like that second string I have in my original post. Do you think JSON + populating the LZW dictionary would be better than just sending less data through LZW? It would make it easier to handle and modify Dec 29, 2023 at 20:23
  • 4
    +1. The idea of delta compression is useful, but risks more errors (c.f lost i-frames). Lost baseline implies undecodable updates. Repeated baselines implies more bandwith. So, a tradeoff. Triggered updates also useful. I think GPS satelites also update at intervals. Regarding errors, consider transmiting/acquiring sensor data as raw as possible, to lessen the impact of systematic error caused by faulty data processing.
    – Pablo H
    Dec 30, 2023 at 23:49
8

Many compression schemes use the fact that consecutive samples are often closely related, and the differences may use less bits than original data. If you transmit one full data packet followed by a number of incremental packets the average bits per packet may be much lower.

4
  • This is actually a good idea and the way that some of the sensors (accelerometer, magnetometer) output data, that actually might be easier. Dec 29, 2023 at 20:17
  • 3
    That’s clever, however it’s important to consider the risk associated with the loss of a packet. All subsequent packets will then be unintelligible/unreliable, so regular “full paints” should be included to mitigate.
    – Seb
    Dec 30, 2023 at 2:02
  • 3
    Many video codecs do exactly this. The "full paints" are called key frames. The challenge for adapting it to this would be deciding how many bits smaller your delta fields should be. Dec 30, 2023 at 4:33
  • I'd send the deltas against the key frame, not the most recent delta frame. Probably slightly higher bandwidth, but if you lose a delta frame, that's the only frame you lose. (If you lose a key frame, you lose the entire sequence either way.)
    – Mark
    Jan 2 at 1:14
5

There are many binary formats for structured data. Wikipedia has a list. Some examples are EXI, CBOR, FlatBuffers, Protocol Buffers.

The are simpler to implement than a compressor, and simpler to decode, and probably compact enough for simple binary data (such as a bunch of floats or halfs).

Using a standard (container) format has the benefit of allowing for off-the-shelf code, interoperability, less risks (e.g. versioning), archivability, etc.

Update: There are even tools to describe, (de)serialize and visualize formats. Kaitai Struct is such a tool, and they have a concise comparision of a good part of the serialization "design space".

4

Your json has an advantage over binary. Things can be left out. If, for example, the Accelerometer has nothing new to say durring this sample you can leave it out.

However, if GPS, Accelerometer, temp, Pressure, Magnetometer, Accelerometer, Humidity are going to show up each and every time, each with exactly the same number of fields then text isn't doing you much good. All it's doing is letting you read it with an editor. What you have is a fixed record. And those encode into binary very nicely. You don't have to keep telling us that 3rd, 4th, and 5th numbers are from the accelerometer because they're always from the accelerometer. That lets you completely remove all the meta information about what each number is and just send the numbers. You can document what each is at your base station. Meta info doesn't need to fly.

That saves a lot of data right there. You get a little bit more compression turning text into "binary" (floats, ints, etc). Things can get lossy here though so follow Barts advice and look into the original forms this data took. Lossy isn't always bad. But you should always be aware when it's happening.

You can look into what a canned compresion algorithm can do for you. But if your record is always the same data points try this and measure it against the canned compresion algorithm. You might be suprized by how good you can do without it.

Bit packing can squeze more data in but remember to end on a byte boundry. Don't kill yourself shoving numbers together if it doesn't save you at least 8 bits a record.

Also, be aware of the law of diminishing returns. This hole is bottomless. Always check if what you're getting out of this is worth what you're putting into it. Or you'll never stop optimizing.

but if I wanted to add more sensors

Add them at the end. This is called single data inheritance. This way the position of the old numbers doesn't change and can be read by old code.

more accurate sensors

Oops, if this changes the number of needed bits you'll have to change the format and have to write new code to read it. This issue creates the need to express which format your data is in somewhere.

And that brings me back to the advantage json has over binary. Not only can things be left out, they can be added. Compressed json can be decent. It's very flexible, and maintenance friendly. But sooner or later every coder tries hand rolling a compressed format. It's good to see for yourself what makes a difference and how much difference it makes. Keep a close eye on what you're losing as you pursue this.

6
  • I am not actually using JSON in my code, that was there so it was a little bit more "human readable" for this question, the second compressed string is closer to the data format that I send back to the base station. It's pretty much the same but it assumes the order of the data so I can drop the header information, but you are right this could cause issues for me when I change the payload. I'm also not too concerned with whether the data is lossless because the actual research data gets written to a disk onboard. But things like GPS would need to be pretty accurate so we can find the payload. Dec 29, 2023 at 20:26
  • You can fix the problem of needing to skip values by putting in invalid sentinals. For example, GPS goes from [-180, 180]. So use -255 as the value when there is no value. The overhead of JSON, or a JSON-like schema is always going to be an order of magnitude more than the extra sentinels sent down. For GPS, you can fit the entire lat or long in 64 bits. There's a reason why hardware never uses text. Dec 30, 2023 at 2:34
  • 3
    Binary doesn't necessarily require sending everything! You can have several message types, one for each set of pieces of information: GPS message, Accelerometer message, etc... it'll take a single byte of overhead per message to have up to 255 messages, or you can shave it down to 4 bits for up to 16 messages. Dec 30, 2023 at 12:01
  • 1
    @MatthieuM.: Or one could have eight types of information and use one byte per message to specify which ones are included, or two bytes to specify up to 16 types, etc.
    – supercat
    Dec 31, 2023 at 21:20
  • @supercat: Indeed, that's another possibility. For separation of concerns I favor splitting messages, but it does generally have (slightly) more overhead than a bitset presence header. Jan 1 at 10:20
3

Consider zstd for compression. It has custom dictionaries, besides many other good things. Killer feature is you can train a dictionary on all those "JSON" packets collected earlier, getting a highly-tailored dictionary

1

Some interesting answers here, but as far as I can see, any of them misses the elephant in the room: you are already using an LZW compression algorithm, which is eliminating most of the redundancies in your data.

This can mitigate a lot of any of the suggested micro-optimizations of the input data size - when the input data is already a highly packed binary, LZW will not compress as much as when the input data is a not-so-highly packed JSON.

Of course, that means when you find ways to reduce your input data size by, lets say, 30%, the LZW compressed data size will still yield to a gain between 30% and 0% in output size, but I would expect it way more near to 1-5%% than to 30%.

One point which might be worth a look is your own LZW implementation. I made the personal experience that own implementations of these algorithms will rarely beat existing, popular compression libraries/tools, which have been optimized over decades. For example, did you try to use the LZMA SDK of the well-known tool 7zip? It is highly portable, has a liberal license and is one of the best general-purpose compression algorithm implementations I know.

1
  • I did know about the 7zip package, but I kind of wanted to learn more about the algorithm so I wrote my own implementation but its honestly pretty bad, so moving over to that SDK is already on the roadmap haha. I kind of figured "just use an existing compression algorithm" would be a suggestion but since I am already using (as far as I can tell, and ignoring my bad code) one of the best compression methods for binary data, I wondered if there was ways I could optimize the data before compression. Dec 29, 2023 at 20:16
1

Multiply each number by a fixed number of decimals to remove the decimal point. Encode each in a fixed number of binary bits big enough for every case. If you need ASCII then use base-85 encoding.

If the format changes add a 6 bit field for one of 64 data formats at the start.

If you send data 10 times per second then you can send the difference to the previous set, saving quite a few bits. Send the complete data once per second.

-2

It strikes me as odd that someone who implements his own compression to minimize payload uses text to represent his raw data. But to be fair, given this pipeline, you are not likely to gain much from starting with a compact binary record.

Another thing that puzzles me is that I don't see any timestamps. Is it all realtime and does the receiver add the timestamps based on the moment of reception?

I would focus on getting faster and more reliable transmissions, software does not seem to have much to offer here.

Not the answer you're looking for? Browse other questions tagged or ask your own question.