5

We have a microservice that reads in large data files, each row containing information about Accounts.

For each record found, it will then go off and do other "stuff" and that stuff is distributed across other microservices. As such, we need to look at using correlation IDs to tie distributed transactions together from a logging perspective.

We're struggling understanding the boundaries of those correlation IDs though.

I'm aware that the general principal is "use a correlation ID you're given or create one", but it's the singularity of that correlation ID that's giving us the issue.

For example:

  • data file contains 1,000 Account records - it would be useful to pull back the logs of all the activities against that 1 data file across the 1,000 Accounts; but
  • it would also be useful to just pull back the resultant logs for a single Account in the data file.

So it feels like two separate Correlation IDs are needed - one for the file, one for the Account - in a stack, almost.

However nothing I've ever seen on the interwebs ever approaches correlation in this way, so I'm guessing my thinking / design pattern is wrong.

Any tips? How have others dealt with this?

1
  • 1
    Alternatively you could compose both information into one. Apr 9, 2018 at 7:35

4 Answers 4

1

I have been thinking a bit about this question and I think I found a way to get composed IDs, meaningful enough for us to search and correlate files, accounts and transactions.

account_id@sercive_name://file_path/file_name.ext

It's basically a URI. Meaningful enough for us to know the account, the file where It came from and the service involved in its processing.

Now, we would have to decide where to put the correlation ID that identifies the whole transaction.

It could be at the query string

account_id@sercive_name://file_path/file_name.ext?transaction=ID

Or it could be part of the credentials *

account_id:tx_id@sercive_name://file_path/file_name.ext

Services only need to replace service_name by its own name (or any kind of service code we might have).

We could add even more information, as for instance the date of the process

account_id@sercive_name://file_path/file_name.ext?transaction=ID&intake=yyyy-MM-ddTHH:mm:ss+00:00

In distributed systems, could be interesting to know the instance of the service too. For example, as a subdomain

account_id@sercive_name.instance.server_name://file_path/file_name.ext?transaction=ID&intake=yyyy-MM-ddTHH:mm:ss+00:00

No need to say that we could swap the position of the elements. For example

tx_id@sercive_name.instance.server_name://file_path/file_name.ext?account=account_id&intake=yyyy-MM-ddTHH:mm:ss+00:00

Or

tx_id@sercive_name.instance.server_name://accounts/{id}?file=file_name.ext&intake=yyyy-MM-ddTHH:mm:ss+00:00

Just find out which one suits your needs of querying. Just remember that the more info It has, the less chance for collision and more information is retrieved per query.


* Honestly, the last one doesn't like me as much as the first. If I had to choose, I would go for the first.

0

You can prefix the correlation id for the file. You would generate a correlation id for the file, then when you start processing each individual row update the correlation ID to append a new suffix. This makes some assumptions about your ability to control the correlation ID, and search for the prefixes.

1
  • Thanks - you've identified the reason I've not gone down this road just yet - the support for the tooling to use prefixes.
    – Chris Wood
    Feb 8, 2018 at 9:57
0

create a "." separated correlation id and put whatever level of information you want to put, then somebody who takes the log from diff. microservices split the correlation id based on "." and provides the view you want

0

There's a Zipkin convention for doing just this, which uses a trace-id, parent-span-id and span-id. The highest level (thing which initiated everything) creates a trace-id with a span-id having the same value as the trace-id. It has no parent-span-id. This could be in your case the file. Each downstream unit would have the same trace-id value and a newly generated span-id value with the parent-span-id value set the the initiating span-id. There can be further nesting of span-ids. Although Zipkin is meant for use across network boundaries with specifically named headers, I have also used it in a single application process to delineate parts for logging. Quicklog.io is a traceable event log service which might interest you. Here's some info on the Zipkin headers https://github.com/openzipkin/b3-propagation

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.