Scoping of Correlation IDs in microservices

Question

We have a microservice that reads in large data files, each row containing information about Accounts.

For each record found, it will then go off and do other "stuff" and that stuff is distributed across other microservices. As such, we need to look at using correlation IDs to tie distributed transactions together from a logging perspective.

We're struggling understanding the boundaries of those correlation IDs though.

I'm aware that the general principal is "use a correlation ID you're given or create one", but it's the singularity of that correlation ID that's giving us the issue.

For example:

• data file contains 1,000 Account records - it would be useful to pull back the logs of all the activities against that 1 data file across the 1,000 Accounts; but
• it would also be useful to just pull back the resultant logs for a single Account in the data file.

So it feels like two separate Correlation IDs are needed - one for the file, one for the Account - in a stack, almost.

However nothing I've ever seen on the interwebs ever approaches correlation in this way, so I'm guessing my thinking / design pattern is wrong.

Any tips? How have others dealt with this?

Answer 1

I have been thinking a bit about this question and I think I found a way to get composed IDs, meaningful enough for us to search and correlate files, accounts and transactions.

account_id@sercive_name://file_path/file_name.ext

It's basically a URI. Meaningful enough for us to know the account, the file where It came from and the service involved in its processing.

Now, we would have to decide where to put the correlation ID that identifies the whole transaction.

It could be at the query string

account_id@sercive_name://file_path/file_name.ext?transaction=ID

Or it could be part of the credentials ^*

account_id:tx_id@sercive_name://file_path/file_name.ext

Services only need to replace service_name by its own name (or any kind of service code we might have).

We could add even more information, as for instance the date of the process

account_id@sercive_name://file_path/file_name.ext?transaction=ID&intake=yyyy-MM-ddTHH:mm:ss+00:00

In distributed systems, could be interesting to know the instance of the service too. For example, as a subdomain

account_id@sercive_name.instance.server_name://file_path/file_name.ext?transaction=ID&intake=yyyy-MM-ddTHH:mm:ss+00:00

No need to say that we could swap the position of the elements. For example

tx_id@sercive_name.instance.server_name://file_path/file_name.ext?account=account_id&intake=yyyy-MM-ddTHH:mm:ss+00:00

Or

tx_id@sercive_name.instance.server_name://accounts/{id}?file=file_name.ext&intake=yyyy-MM-ddTHH:mm:ss+00:00

Just find out which one suits your needs of querying. Just remember that the more info It has, the less chance for collision and more information is retrieved per query.

---

^{* Honestly, the last one doesn't like me as much as the first. If I had to choose, I would go for the first.}

Answer 2

You can prefix the correlation id for the file. You would generate a correlation id for the file, then when you start processing each individual row update the correlation ID to append a new suffix. This makes some assumptions about your ability to control the correlation ID, and search for the prefixes.

Answer 3

create a "." separated correlation id and put whatever level of information you want to put, then somebody who takes the log from diff. microservices split the correlation id based on "." and provides the view you want

Answer 4

There's a Zipkin convention for doing just this, which uses a trace-id, parent-span-id and span-id. The highest level (thing which initiated everything) creates a trace-id with a span-id having the same value as the trace-id. It has no parent-span-id. This could be in your case the file. Each downstream unit would have the same trace-id value and a newly generated span-id value with the parent-span-id value set the the initiating span-id. There can be further nesting of span-ids. Although Zipkin is meant for use across network boundaries with specifically named headers, I have also used it in a single application process to delineate parts for logging. Quicklog.io is a traceable event log service which might interest you. Here's some info on the Zipkin headers https://github.com/openzipkin/b3-propagation