1

I have a dataset stored in ElasticSearch, only for searching purposes (the source of truth is external). Each index have an associated schema and there are associated queries.

From time to time, the structure changes from time to time the structure change, doing that, I update all the indexes, by dropping them and recreating them from the source of truth (with the new schema definition). This update takes place in the background, so code still fetch old format data. To mitigrate that, I have a layer which deal with data format compatibility issues.

The issue is that the requests target the new data schema, breaking on the old dataset. How do you handle data migration at this level?

0

I Haven't done it in elastic, but look into "Schema on Read"

https://www.elastic.co/blog/schema-on-write-vs-schema-on-read

the general idea is that you hold unstructured data and then define your fields as part of the query.

This helps with schema changes because you can write a query which can use a json path, regex or expression to extract data from more than one possible location and coalesce it into a single field for your report or graph.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.