4

In our system a media file can have several states:

  • invalid
  • empty
  • uploading
  • ingesting
  • ready…

..depending of the state, information about the media is accessed in different ways. For example:

  • During upload the file name is on a temporary upload info JSON file on the filesystem.
  • When the object is ready this file name would be guessed by looking at a specific place on the filesystem (namely under an directory named after the ID of the media).

Same goes for the "progress" attribute of a media :

  • During upload "progress" represents the downloaded bytes / file size.
  • During the ingestion step, the progress represents the current step number / total number steps (thumbnailing, post-processing, etc).

The serialized state of a media is used by the UI to feed information back to the user.

A MediumStateFactory tries a bunch of AbstractMediumState derived classes (MediumStateReady, MediumStateUploading, etc) until one qualifies to handle the current state of the media.

Turns out that for tiny files the state can change during serialization: an MediumStateUploading can be built at the beginning of the serialization, but as the state of the media changes, by the time the actual serialization is done it doesn't represent the media anymore, leading to errors. For example, a MediumStateUploading expects to retrieve the media size from the temporary JSON.

Currently we are restarting the serialization entirely if we catch an exception during the process. This is rather brute force and quite ugly.

What would be the best way to serialize objects that can mutate during the serialization process?

  • What attributes does a MediumState have? Does it fetch all attributes lazily from the persistent storage (which location changes during the processing of the media) or does the factory load them once and the object keeps them in RAM after that? – 5gon12eder May 16 '16 at 20:16
  • This reminds me of data bases that can give you a consistent, unchangable view. Maybe you can borrow solutions from database implementations. – User Aug 1 '16 at 22:34
2

I realize that your question is specifically about serialization, but I think you may be trying to solve the wrong problem here. I suggest you to think about whether you have separated concerns correctly in your application design. It may make more sense to consider the serialized state of a media file as a composition of some unique ID and some possibly outdated state information, and use the latter only for user information. You could then delegate the real bookkeeping based on the unique ID's to an entity that does not have to rely on the possibly outdated state information coming back from the UI.

2

That's the problem with mutable objects, that they can mutate.

Quite obviously, if you managed to correctly serialise the state that your object has right now, it won't be the state that your object will have two seconds from now. So having the serialised state not match the current state must be acceptable, otherwise you have lost.

If the changes to your object are independent, and it is acceptable to serialise your object with any set of complete changes (for example the set of files stays unchanged, but the state of each file could change), then the easiest would be to have a lock for each file description, and you look a file description around its serialisation, and around modifications.

If you have suitable threading, you might dispatch the serialisation and the object modifications to the same serial thread, which would make locking unnecessary.

2

Separate writes and reads. Below is a way to allow both to occur at the same time to the same file.

Maybe you need for writing or changing the state of the media. It has a list of writers and a parallel list of states. If the write fails beacuse it violates an invaraint, change the state. It would notify observers list with the state.

The medium state factory is an osberver. It has a cache of the latest state. There is a parallel array of state and abstract factories. If the latest state is non changing it unknown, delgate to the original method. Otherwise, find the index in the arrays and use that abstract factory.

0

Is it possible and feasible to split the serialised data into parts? One part containing the stable part of the state and one small part containing the changing part of the state.

This way you have several options:

  • you can serialise the small changing part last;
  • you can use events, observer pattern or reactive paradigm to serialise the changing part of the state when a change occurs;
  • if the serialisation is for persistence, you might be able to omit the changing part or append it to a sequential transaction log; or,
  • you can restart serialisation of only the changing part just as you do now.
  • What about streaming the process. If serialize is not for persistence and it's just for monitorize the process from GUI, may be he could redirect the solution to realtime. Streaming changes. Does this make any sense? I'm not sure at all if it's a good alternative. What changes is one outputstream (file) by another different – Laiv May 16 '16 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.