Can anyone point me to patterns for, or material on decentralised federated system design / federated data/service architectures?

To give some more background on the problem I'm looking to solve, here's the full context. I've had an idea floating around for a while to create a "reverse address book". i.e. Rather than me having a list of all my friends, their phone numbers, addresses, birthdays, etc, I just take responsibility for managing my own address book entry. I can then connect to my friends to see their entries (or whatever parts of their entries they're happy to share with me). That way, if I move house or get a new phone number, I update my record and everyone I've given permission to see that information sees my record automatically updated in their address book.

The above is simple if we're talking about a single system... But this idea's big (i.e. it solves a lot of GDPR issues, it potentially means that I no longer have to contact 50 different companies when I move house to tell them I've moved, companies can save a fortune wasted trying to track down those who've not given them this information, etc.).

As such, it seems unreasonable to have one system which holds & controls all of this information. Security concerned users may want to run their own instance which holds their information; but still enable friends using the "status-quo implementation" to access their data. Alternatively, other companies may want to host their own flavours of such a system, so it's not a monopoly; but again we'd want users of any implementation of this solution to be able to see other users even if they're on a different implementation (in the same way you can use any email provider and still talk to friends on other mail providers).

Apologies that this is quite a loose question; I've tried searching but don't know enough terminology in this area to know the correct search terms, as all systems I've worked on to date have been centralised, or at most had a master database with multiple slaves, rather than being fully decentralised / federated.

I've come up with a few ideas on this for this:

  • Having a centralised register, but leaving everything else decentralised. So any system providing this functionality adds their URI to the register after which all systems can see it... However I don't like this solution as if possible I don't want any centralised dependency.

  • Have each implementation holding a list of approved providers (i.e. for any big companies hosting public solutions), and allowing users to add "ad-hoc providers" to their personal "trusted provider" lists to cater for special cases (e.g. friends running their own personal instances / companies running in house instances). Essentially a similar pattern to how SSL certificates are currently handled in browsers; i.e. the recognised CAs are provided by your browser's vendor (some of these lists being published for general usage), but you can always add your own CAs to your browser as the need arises.

  • 1
    First of all, this is an actual great idea ! Second, have you consider using blockchain ? Jan 19 '18 at 22:58
  • First, I am not sure I get an idea. To connect to a friend, you need to either maintain his entry in your address book to connect to his data service, or at least maintain his ID in your address book to connect to a centralized storage. Either seem to kill an idea. Second - patterns are ways to solve common problems. Your problem does not seem common. Don't look for patterns to solve it. Think about solutions, not patterns. Jan 20 '18 at 8:54
  • 1
    I've been thinking about something quite similar not so long ago. Do you have a website/github for the project?
    – doubleYou
    Jan 20 '18 at 11:58
  • 1
    @lee-pai-long the idea is intriguing. Can you elaborate?
    – doubleYou
    Jan 20 '18 at 11:59
  • @lee-pai-long; thank-you. Hadn't considered it; have seen a lot about it in the news, but don't know much about it... will take a closer look.
    – JohnLBevan
    Jan 20 '18 at 20:11

Let's break down the problem:

  • You need some way to identify users
  • Given an identifier, you need to have some way to determine how to push/pull information

Additionally, you obviously need some kind of standard protocol and a way to manage access rights. Also, the user id should be something people can remember, not some IPv6 address or the like.

The E-Mail Approach

In the end, you don't get around some centralization. But you may be able to delegate this part to already established infrastructure.

Example: E-Mail, which has similar problems, solves both of these issues by including the host information (mail server) in the user identifier (address). While e-mail itself is quite distributed, it relies on DNS to resolve host names, which is - at it's root - centralized.*

You can go a similar way. Either come up with your own naming convention (e.g. providerXY.com>>AlbertEinstein79) or just use the e-mail format. You can even go one step further and actually use e-mail for your system's communication, i.e. you don't define a protocol but an e-mail template that the end-points can parse - not that I would necessarily recommend this route.

One downside of this approach is that it ties the user id to the underlying host, which means you can't really move your account without creating a new identifier.**

* The centralization comes from the fact that everyone recognizes a few servers as authoritative.

** If your system is push-based, you could use it to notify subscribers about your new account. However, this doesn't work if someone has written down your id on paper and tries to access it after your move. Anyways, push based is probably not the way to go.

Lookup Approach

If you want to decouple the host from the user identifier (so that you can switch hosts), you need some kind of lookup service. If this should be decentralized, there are some problems to be solved.

  • Ensure uniqueness of IDs
  • Ensure that only the owner of the ID can change the host
  • Ensure that one person cannot just create a couple billion IDs

While I'm not sure about (but open to) storing the actual data in a blockchain, it would actually make a good lookup store. I'm not really an expert on blockchains, so I'll refrain from considering the details here, but it seems to me the above mentioned problems might be solvable this way.


First @VladislavRastrusny is right.

This is not a pattern problem, a pattern is a general abstract way of solving an abstract known problem, that you then (try to) apply to a concrete case.

Here @JohnLBevan you seems more to have difficulties transforming an app/service/platform idea to "reality".

So to follow my comment and elaborate:

Blockchain at is core is a decentralized, distributed and hard to temper(secure to some good enough extend) database system.

If every user as his own block with his informations, and the "app" let him(or other service/app on his phone/computer automatically) search for other people information.

The "link" or "transaction" part can be used to give access to information, so in our situation if Bob authorize Alice to have his informations we have transaction et Bob and Alice are linked.

Blockchain as a system call PoW(Prove of Work) and can guarantee that only Bob is able to alter his information, therefore everybody in the network can be pretty confident that they'll not be another person pretending to be Bob.

Bob is only responsible for his entry and the validity of it. When Alice wants to, for example, call Bob, the app search for the Bob block and if it is considered valid(through PoW) by the system, transfer the phone number to the dialing app, and voila.

Of course this only works if Bob has an up-to-date entry but this problem is already here, and by having to maintain only your entry, the cost is fairly low, that's the all point right.

This is why your idea is great, because your app can become a decentralized, distributed IAM system(you'll need to secure the access to users devices with MFA).

Note that there is an alternative to Blockchain called Hashgraph you may want to look into that too.

Anyway, it will not be easy by any mean, so you'll need to surround yourself with the best people you can.

But again it seems like a great and ambitious idea, so I wish you good luck.

  • Thank-you. FYI: I was looking for a pattern not for the whole system, but for the specific problem of having data spread through disparate systems, and having a way to query them without having to broadcast every query to every system; something which I assume has been an issue for other federated / decentralised systems in the past, so for which someone may have defined a template solution / pattern... I don't think I explained that very well in my question though.
    – JohnLBevan
    Jan 20 '18 at 20:27
  • Blockchain and Hashgrah do look promising for this; will read up on how those work in more detail; thank-you for pointing me that way.
    – JohnLBevan
    Jan 20 '18 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.