Dealing with queries on versioned data

Question

I have a dataset stored in ElasticSearch, only for searching purposes (the source of truth is external). Each index has an associated schema and there are associated queries.

From time to time, the structure changes. When doing that, I update all the indexes by dropping and recreating them from the source of truth (with the new schema definition). This update takes place in the background, so code still fetches old format data. To mitigate that, I have a layer which deals with data format compatibility issues.

The issue is that the requests target the new data schema, breaking on the old dataset. How do you handle data migration at this level?

Answer 1

I Haven't done it in elastic, but look into "Schema on Read"

https://www.elastic.co/blog/schema-on-write-vs-schema-on-read

the general idea is that you hold unstructured data and then define your fields as part of the query.

This helps with schema changes because you can write a query which can use a json path, regex or expression to extract data from more than one possible location and coalesce it into a single field for your report or graph.