0

I have a rails app where users can upload data and then the backend processes it and creates a bunch more data that gets stored with the record. The data processing is async and handled outside of the rails app. Currently I have been storing the unprocessed data in the same table as the processed data and then adding the rest when it is processed but I want to stop the rest of the app touching the data before it's finished processing and also make all the columns on the table not null.

The simplest solution I can think of is creating a new table for the data to be processed and moving it over when it's ready but because this data will only be sitting unprocessed for a maximum of a few minutes it doesn't really make sense to store it on disk and then delete it shortly after.

The data itself is just some strings, ints, timestamps and an XML file attachment.

What methods would be good here for storing this temporary data? Just as a guess I think Redis might be good for the job but I haven't used it enough to know properly.

  • If you don't want to give up the benefit of NOT NULL constraint (and I think that's a good idea), consider splitting the final records into two tables (user-contributed data and computed data) with a foreign key relation between them. That saves you the effort of copying things around and keeps all integrity constraints (at the price of retrieving two records instead of one for later reads). – Kilian Foth Aug 30 '18 at 6:29
4

If you don't want to give up the benefit of NOT NULL constraints (and I think that's a good idea), consider splitting the final records into two tables. One holds the data explicitly contributed by the user, the other holds computed data and a foreign key to the first table. That saves you the effort of copying things around and keeps all integrity constraints (at the price of retrieving two records instead of one for later reads).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.