We are loading in insurance claims data into Parquet on a daily basis for analysis in Spark. At a high level, each claim record has a date, claimant details and value of the claim. This works well in Parquet because we have partitioned the data by date, so we can treat the Parquet files as immutable and just write to a new daily Parquet file for each daily load.

However, we would now like to process corrections to historical claims records. For example, sometimes a claim may be reversed several days or months after we have loaded a claim record, or an incorrect value in a claim record may be corrected. The number of corrections is very small compared to the total number of claims.

The problem is that to apply the corrections in our current schema, we'd need to go back and rewrite our historical Parquet files with the corrected records. Unlike in a traditional SQL database, we can't just go and update individual records. If a single daily data load contains 100 corrections, it may mean we need to rewrite up to 100 Parquet files, which is going to be an expensive operation.

Is there a general Parquet design pattern or suggested approach for handling this type of situation? I imagine it is quite common in a lot of different domains.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.