17

I am wondering if serialization and deserialization of classes should always be treated as an "atomic transaction?"

What I mean is, if an error were to occur during the process of serializing or deserializing a member of an object, should the whole serialization/deserialization of the object be considered to have failed?

For a more concrete example, I am going to use C++. Suppose a very basic structure as follows:

struct RGB
{
    uint8_t r;
    uint8_t g;
    uint8_t b
};

Suppose I have an RBG instance defined as so:

RGB myRGB{0x10, 0x20, 0x30};

Which, if serialized into a raw binary stream would look like:

0x10 0x20 0x30

Suppose, the one of the bytes gets lost during transmission so that the "deserializer" is fed only:

0x10 0x20

I can see two options here.

a) Because the 'b' member of the struct cannot be deserialized, the whole struct cannot be deserialized.

b) 'r' and 'g' can be deserialized, and will we just use the default value for 'b'

Both have their merits. The problem I can see with (b) is that, while it ensures you at least get "something" it is not actually an accurate reconstruction of the thing that was serialized which (for a more complex example) could result in further errors down the line.

I suppose an option (c) would be:

c) It depends on the application. If the object in question can be default constructed, then option (b) is fine. If the object cannot be default constructed (i.e., requires values in its constructor), all of the values required for constructed need to be deserialized atomically.

Improve this question
10
  • 8
    It would, of course, be quite useful to know that an error occurred that might have caused data loss. That communication can take place a number of different ways.
    – Robert Harvey
    Feb 13 at 17:30
  • 37
    Transmission loss is a separate concept from failure to serialize, each of which is detectable in its own right (e.g. checksum/signature vs. exception).
    – Erik Eidt
    Feb 13 at 19:26
  • 5
    Lots of answers already, and I agree with the general "almost never", but you might want to consider a couple of cases: data versioning (where the new format has more data but you still want to read old format, and you can't otherwise tell them apart) and "progressive enhancement" (e.g. progressive JPEG) where data is being transmitted over a slow serial link and you want to make sense of an incomplete transmission.
    – pjc50
    Feb 14 at 16:27
  • 6
    How would you know that 0x10 0x20 is the R and G channels? It could be possible that a G value of 0x15 went missing in the middle and what you have is actually the R and B channels? Or does this question assume that the data loss occurs in e.g. a truncated file, where it will only ever miss data from the end?
    – Luke
    Feb 15 at 20:37
  • 4
    @Luke You don't know... Which I guess is another good reason as to why you should NOT try to default some values. You could end up with a wrong value in some field AND a default values (so 2 errors resulting from a single missing byte).
    – Patrick Wright
    Feb 15 at 20:43

10 Answers 10

Reset to default
38

Meyer's DbC design by contract has something to say about this.

Suppose that one of the bytes gets lost during transmission so that the "deserializer" is fed [just a prefix of the bytestream before EOF truncation].

We are constructing an RGB pixel object. That class makes certain guarantees: values shall always be integers in the range 0 .. 255 and never e.g. a NULL pointer or a NaN value.

The .deserialize() method makes a guarantee: it shall return a valid RGB object that was received from the bytestream.

If it can't do that, it should return nothing at all -- it should raise an exception to indicate it was unable to fulfill its contract.

How the calling code responds to that is up to it. Things are out of the hands of .deserialize once it did its job.

Improve this answer
5
  • 10
    Nitpick: Since the three members are declared uint8_t, it's generally plain impossible for them to contain anything other than numbers in the range 0-255. Those 256 numbers correspond one to one to the 256 possible bit patterns in eight bits. I.e, there is no pigeon hole left for a NULL or NAN value.
    – cmaster - reinstate monica
    Feb 14 at 20:07
  • 11
    Yup, yup, I'm aware. I was giving a multilingual response. In a language that can raise exceptions, DbC offers good guidance for keeping your promises. In a dynamic language there's a temptation to substitute a non-int type to show that an exceptional thing happened, and then caller will misinterpret it. In the C++ language some folks will be tempted to declare that RGB only goes to 254, with 255 being exceptional, also leading to caller misinterpreting it. The key is to preserve class invariants and only return correct answer values. If the ctor gives (0, 0, 0) and we hit EOF, don't return it
    – J_H
    Feb 14 at 20:13
  • 3
    Note that raising an exception is not the only way to signal failure, even in C++. The deserialize function could return std::expected<RGB>, or something similar.
    – Matthieu M.
    Feb 15 at 8:03
  • 4
    @MatthieuM. if the contract has std::expected<RGB> as return type, then it returning an unexpected object doesn't constitute a failure of the contract, as the contract explicitely states that possibility. To be consistent, such a function should probably be called TryDeserialize as it's expected for it to not succeed in the operation, but that still amounts to success for what the contract says.
    – bracco23
    Feb 15 at 14:48
  • 3
    @bracco23: There are 2 contracts here, the contract (invariants) of RGB, and the contract of the function. The failure I referred to is about the contract of RGB (not fully constructed), and yes I do agree that TryDeserialize would NOT be failing its own contract if it fails to deserialize due to incomplete input.
    – Matthieu M.
    Feb 15 at 15:51
35

I would say it is c) or something similar to c): "it depends on the application and the data that is being transmitted".

If I am serializing the details of a banking transaction, the idea that you would accept a partial transaction and fill in the missing details seems pretty crazy, right? It should because that could land the institutions involved in some hot water.

I would go as far as to say that, by default, all transmissions should be considered atomic and only once it is determined that a partial transmission can be tolerated, that you would implement any other solution. Generally speaking it's only going to be things that don't matter that you can treat non-atomically. For example, if you are sending out a list of new cat gifs available on a website, maybe you just want to show what you received instead of nothing.

Improve this answer
5
  • 16
    This is a very helpful statement: "all transmissions should be considered atomic and only once it is determined that a partial transmission can be tolerated, that you would implement any other solution."
    – Patrick Wright
    Feb 13 at 19:38
  • 9
    There is a second possible reason for not failing the deserialization: When the data is particularly valuable (think: downlink data from a voyager probe, or similar), it would be a good idea to implement the notion of a missing value into the data model.
    – cmaster - reinstate monica
    Feb 14 at 20:12
  • 12
    @cmaster-reinstatemonica In a scenario like that, I would probably store the raw data as is before attempting to parse it. That way if you can't deserialize it for any reason, you don't lose information. But yes, that sounds like a valid use case.
    – JimmyJames
    Feb 14 at 21:26
  • 1
    For an example of "partial transmission", some photo-album software I'm working on can track camera position in the form of (latitude, longitude, altitude, heading, pitch, roll). A position is considered valid if it has a valid latitude and longitude: if any of the other values is invalid, that particular value gets marked as "unknown", but if either the latitude or longitude is invalid, the entire position gets marked as "unknown".
    – Mark
    Feb 16 at 1:53
  • 5
    The question usually boils down to "what is the smallest useful atomic chunk of data". A single RGB pixel is either complete or not. But for example a stream of logs or audio data, can usually be separated into logical chunks, where one still makes sense without the others. So each log line, or each unicode-character or each frame could be the atomic part.
    – Falco
    Feb 16 at 10:54
29

Don't tell lies to your caller. He relies on the results being correct.

Returning an object from a deserializer tells your caller that this was the content found in the serialized data.

If you can't reliably reconstruct the original object, inform your caller about that exceptional situation. In that error message (exception object?), you might choose to include a best guess about the contents, but you should not disguise such a guesswork as being the truth.

If you are sure that, from current problem analysis, it's also acceptable to e.g. replace some missing fields with their defaults, be sure to document this highly unexpected behaviour in a place where every future user of your deserializer will take notice.

Improve this answer
1
  • document (..) in a place where every future user of your deserializer will take notice I'd add that actually there are only two such places that every future user will notice - the name of the function and the return value type (in statically typed language, if they use it wrong). The name doesn't have to be too descriptive, but it should indicate shenanigans. deserializeWithDefault, deserializeWithFallback, deserializeOrGuess. If relying only on return value type, it should fail if used directly as the "expected" type.
    – Frax
    Feb 17 at 12:48
6

This is highly contextual, both from a business perspective (as JimmyJames mentions, an partial banking transaction) but also from a data model perspective.

If you're dealing with sequential individual elements, such as:

A, B, C, D, E, F, G

Then you can reasonably argue that each individual element's parsing can be individually decided (assuming no business conflict like was mentioned before).

However, if your data is nested, such as JSON or XML, then you inherently can never fully process an element since the root element is only closed at the end of the transmission. If your parsing fails halfway, you cannot have parsed the root element correctly, and therefore you cannot decide whether the information you did parse is individually complete enough or not.

Again, context applies. Maybe your root element is really just an array of discrete items, at which point the earlier advice might apply. Similarly, maybe even an array of items is only meaningful when fully known, at which point the earlier advice doesn't apply.

In short, there is no absolute here. If you have to err one way, I would err to assuming atomicity, simply because the impact of making a wrong call is significantly less than when you err the other way.

Improve this answer
3

The behavior of a deserialization/decoding routine should depend upon the extent to which its output will be considered a Source of Truth.

If a deserialization/decoding routine's output will be used ephemerally, and it's better to produce mostly-correct output quickly than perfect output slowly (this situation may arise with audiovisual decoders or applications that perform real-time interactive rendering) then producing imperfect output when given imperfect input may be better than simply providing an error indication.

If a deserialization/decoding routine's output may be used as a Single Source of Truth, then the routine should not produce incorrect output. If, because of faulty input, it cannot produce correct output, then it shouldn't produce any output at all.

If a deserialization/decoding routine will be used to try to extract useful information from input that is known or expected to be corrupted, and its output, though imperfect, will be better than any other Source of Truth, and will thus become the new Source of Truth, then it should produce the best possible output from even corrupted parts of the input, but its output should also include information about any problems that it "patched over".

Probably the most important distinction is between the first two usage patterns. In the first scenario, a transient problem that causes the decoder to receive invalid data may result in transient disturbance to the output, but if the output is going to be discarded in any case, the fact that it is invalid won't matter. If, however, a transient disturbance on the input would prevent the data from being decoded correctly, substituting default data from what will be interpreted as the Source of Truth may result in real data being permanently overwritten, in circumstances where refusing to decode anything would have resulted in the real data being retained until the disturbance subsided and the replacement data could be successfully decoded.

Improve this answer
1

If the structure is very big and is passed over noisy environment, it may be partitioned into smaller chunks that are more likely to pass undisturbed than the whole message in completeness. In this case only processing of the single chunk needs to be atomic. If some chunks have not been received correctly, it may be possible to ask re-sending only the failed segments. For instance, a big image could be split into tiles that are displayed when received, and the failed tiles are re-sent and displayed later.

Improve this answer
1

An optimal API would provide both alternatives. This can be implemented by returning the object and a success status (or a list of errors), or alternatively by having a parameter such as ignore_errors.

This is useful when loading user data files, which can get corrupted due to various reasons and may be the only available copy. Being able to provide a "recover as much as possible" option can save someone's day.

In comparison, if the corrupted data is the parameter of a request, it is better just to reject the request and have the other end retry.

In practice though, many deserialization libraries will not provide partial data. The more complex the format gets, the smaller chance there is to recover anything successfully even after a single byte corruption.

Improve this answer
1

Take the JSON file format as an example. If you get a sequence of bytes that are supposed to be a JSON document, you either decode it completely, or you fail completely. A JSON decoder will never say "Here is some partial data, but there are more bytes that I couldn't decode".

That's a very reasonable attitude. It makes your life a lot easier, and prevents disastrous mistakes. In the end, if you are given data that you cannot process or that is incomplete, what are the chances that what you can decode is pure nonsense?

Same with plain http or https. The software that you call will do its best to get the correct results, but the caller will either get a correct result or nothing.

Improve this answer
6
  • A very common counter example is HTML which can be partially parsed and displayed.
    – JimmyJames
    Feb 15 at 15:26
  • 4
    I guess you've never used a streaming decoder? It absolutely will give you a decode of everything before the error.
    – Ben Voigt
    Feb 15 at 15:59
  • @JimmyJames: Showing a broken page does not cause faults. Processing a partial JSON file, however, might (in the right circumstance). Your HTML example is risk-free if anything does go wrong, which makes it a flawed example when you try to derive a more universal answer from it.
    – Flater
    Feb 16 at 4:03
  • @Flater My answer is not derived from this example.
    – JimmyJames
    Feb 16 at 14:27
  • @JimmyJames: I wasn't referring to the answer you posted, but I see how I phrased it ambiguously. My response was only focusing on your comment here.
    – Flater
    Feb 16 at 22:43
1

The general question completely depends on the application. If you are serializing/deserializing a live video stream, you would typically do a best effort and not bother if a few pixels went black. If you are serializing/deserializing economic transactions, you would typically need to detect and flag any anomalies.

If the transfer medium is unreliable, it is normally necessary to add redundant information to be able to detect or correct errors (i.e. add a checksum or its generalization, an error-correcting code). Most media which we regard as reliable employ some mechanism of error detection/correction on top of an unreliable physical medium.

Hence, in the RGB-example, if the transfer medium could be unreliable, the data should maybe not be serialized into three bytes ["r", "g", "b"] but four bytes ["r", "g", "b", "XOR(r,g,b)"] or more.

Mathematically, for a given amount of redundant information, there is a trade-off between detecting errors and correcting errors. Again, it depends on the application whether it makes most sense to detect errors (for flagging/retrying) or correct errors (for best effort/reduced risk) or do a combination.

Improve this answer
1

It should really be atomic. The concept of fail hard, fail fast applies here, especially since serialization and deserialization are usually used at the border of your system, to communicate with the outside world (even if it's storage or DB you have full control over, but especially if network access is involved).

For your particular example, nothing worse than a little off-color pixel may be the result of assuming default values. For other, more involved objects, it may be catastrophic failure later down the road. Slowly corrupting data is one of the worst conditions a computer system can be in. If you forget (or are not aware) that the issue exists, you might hunt down weird bugs in your data forever, wasting days or weeks debugging, building workarounds around it, and so on and so forth. Depending on the data it may also be completely impossible to distinguish correct from bad data, which may make all the data worthless.

It should be very easy to make sure the process is error-free with some measure of certainty by some form of checksum/hash appended to the serialized data.

If you do, on occasion, wish to deserialize known-bad serialized data (for example from a source system you have no control over), then you need to approach that with a lot of sense, and specifically define the cases you wish to accept and "fix" during deserialization. In this case, the operation still needs to be atomic, but you'll define rules to fix the data issues right there and then. In a sense, this is then similar to being error tolerant when, say, parsing HTML or other structured data, with the exception that the serialized form is usually binary.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.