4

I'm not sure how to best approach this problem. I've got an object in my local database, for example this one:

{ title: "my-object", uploaded: false }

I convert this to a file, say my-object.txt and upload it to a server. Once it has been uploaded I set, locally, the field uploaded to true, so it will now become:

{ title: "my-object", uploaded: true }

My problem is how to handle failure? For example, if the application is stopped or crashes between the moment the file is uploaded and the local database is updated, the item will remain with uploaded: false and I will have no way to find out if the file has been uploaded or not.

So what technique can I use to make this process more reliable?

  • You could schedule a recurring "cleanup" process to scan for files and ensure their flags are set properly. But why not leave it the way it is? What problem are you trying to solve? If it makes you feel any better, you could change the name of the flag from uploaded to uploadedAndMetadataSuccessfullyUpdated, but that's kind of a silly name. – John Wu Jun 29 '17 at 22:21
  • Usually you retry a set number of times until both conditions are satisfactory. After said number of failed attempts, report the failure and do some kind of cleanup. – Tulains Córdova Jun 29 '17 at 22:57
  • One could attempt to download the file and compute the hash of the local file versus the one downloaded from the server. If they match then clear the flag. If they do not match or if the file cannot be found on the server, upload it again. – Jon Raynor Jun 30 '17 at 20:08
5

I don't think this is about atomic-operation, synchronization, asynchronous-programming, or transactions.

I think this is about meaningful state.

Right now you have three possible states. The object's uploaded field is true, it's false, or the objects record doesn't exist.

The question then is what do these three states mean? How do we arrive at them? How do we leave them?

I'd like to think we start with the record not existing. Then something decides we need to store a my-object in the DB and now we're uploaded:false. For all I know a decade goes by and then something decides it's time to upload. Since we're still uploaded:false something tries to upload our object. Once the server tells us it has the file we can set uploaded to true.

Now sure anything can happen during all that but there is nothing transactional happening here. We leave uploaded as false until we know it uploaded. If the power cord is yanked out before we manage to flip the bit then we end up thinking we need to upload again. There is no way to stop that.

What we could do is check with the server before uploading and confirm that it needs the file. Or we can blindly overwrite whats there and save the hassle.

What would be silly would be to "roll back" the file upload so we can send it again. All that translates to is having the ability to overwrite the file.

So if those states express everything you need to know then you're fine. Now if you need to do something special or know if an error happened for a good reason like the servers hard drive is full then maybe you want to record more information than true and false somewhere.

Once you decide to do that, think about what you want to do about it.

3

Just change your point of view: interpret the changing of the uploaded field to true as part of the upload operation. If a failure occurs because

  • the upload failed
  • or the application crashed between the moment the file is uploaded and the local database is updated

does not matter. I guess in most real-world cases the second situation will occur so seldom it does not matter if in this situation a file got uploaded twice. The situation is not very different from the case where the file got uploaded to 99% before a network interruption makes the upload fail.

Of course, if the server does not allow upload of an already uploaded file again, then your premise of "no way to find out if the file has been uploaded or not" becomes wrong, in this case you have a way to find out if the file is already on the server, and you could utilize that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.