We have a microservice that reads in large data files, each row containing information about Accounts.
For each record found, it will then go off and do other "stuff" and that stuff is distributed across other microservices. As such, we need to look at using correlation IDs to tie distributed transactions together from a logging perspective.
We're struggling understanding the boundaries of those correlation IDs though.
I'm aware that the general principal is "use a correlation ID you're given or create one", but it's the singularity of that correlation ID that's giving us the issue.
- data file contains 1,000 Account records - it would be useful to pull back the logs of all the activities against that 1 data file across the 1,000 Accounts; but
- it would also be useful to just pull back the resultant logs for a single Account in the data file.
So it feels like two separate Correlation IDs are needed - one for the file, one for the Account - in a stack, almost.
However nothing I've ever seen on the interwebs ever approaches correlation in this way, so I'm guessing my thinking / design pattern is wrong.
Any tips? How have others dealt with this?