Serializing mutating objects

Question

In our system a media file can have several states:

• invalid
• empty
• uploading
• ingesting
• ready…

..depending of the state, information about the media is accessed in different ways. For example:

• During upload the file name is on a temporary upload info JSON file on the filesystem.
• When the object is ready this file name would be guessed by looking at a specific place on the filesystem (namely under an directory named after the ID of the media).

Same goes for the "progress" attribute of a media :

• During upload "progress" represents the downloaded bytes / file size.
• During the ingestion step, the progress represents the current step number / total number steps (thumbnailing, post-processing, etc).

The serialized state of a media is used by the UI to feed information back to the user.

A MediumStateFactory tries a bunch of AbstractMediumState derived classes (MediumStateReady, MediumStateUploading, etc) until one qualifies to handle the current state of the media.

Turns out that for tiny files the state can change during serialization: an MediumStateUploading can be built at the beginning of the serialization, but as the state of the media changes, by the time the actual serialization is done it doesn't represent the media anymore, leading to errors. For example, a MediumStateUploading expects to retrieve the media size from the temporary JSON.

Currently we are restarting the serialization entirely if we catch an exception during the process. This is rather brute force and quite ugly.

What would be the best way to serialize objects that can mutate during the serialization process?

Answer 1

I realize that your question is specifically about serialization, but I think you may be trying to solve the wrong problem here. I suggest you to think about whether you have separated concerns correctly in your application design. It may make more sense to consider the serialized state of a media file as a composition of some unique ID and some possibly outdated state information, and use the latter only for user information. You could then delegate the real bookkeeping based on the unique ID's to an entity that does not have to rely on the possibly outdated state information coming back from the UI.

Answer 2

That's the problem with mutable objects, that they can mutate.

Quite obviously, if you managed to correctly serialise the state that your object has right now, it won't be the state that your object will have two seconds from now. So having the serialised state not match the current state must be acceptable, otherwise you have lost.

If the changes to your object are independent, and it is acceptable to serialise your object with any set of complete changes (for example the set of files stays unchanged, but the state of each file could change), then the easiest would be to have a lock for each file description, and you look a file description around its serialisation, and around modifications.

If you have suitable threading, you might dispatch the serialisation and the object modifications to the same serial thread, which would make locking unnecessary.

Answer 3

Separate writes and reads. Below is a way to allow both to occur at the same time to the same file.

Maybe you need for writing or changing the state of the media. It has a list of writers and a parallel list of states. If the write fails beacuse it violates an invaraint, change the state. It would notify observers list with the state.

The medium state factory is an osberver. It has a cache of the latest state. There is a parallel array of state and abstract factories. If the latest state is non changing it unknown, delgate to the original method. Otherwise, find the index in the arrays and use that abstract factory.

Answer 4

Is it possible and feasible to split the serialised data into parts? One part containing the stable part of the state and one small part containing the changing part of the state.

This way you have several options:

• you can serialise the small changing part last;
• you can use events, observer pattern or reactive paradigm to serialise the changing part of the state when a change occurs;
• if the serialisation is for persistence, you might be able to omit the changing part or append it to a sequential transaction log; or,
• you can restart serialisation of only the changing part just as you do now.