Patterns for Decentralised / Federated Data

Question

Can anyone point me to patterns for, or material on decentralised federated system design / federated data/service architectures?

To give some more background on the problem I'm looking to solve, here's the full context. I've had an idea floating around for a while to create a "reverse address book". i.e. Rather than me having a list of all my friends, their phone numbers, addresses, birthdays, etc, I just take responsibility for managing my own address book entry. I can then connect to my friends to see their entries (or whatever parts of their entries they're happy to share with me). That way, if I move house or get a new phone number, I update my record and everyone I've given permission to see that information sees my record automatically updated in their address book.

The above is simple if we're talking about a single system... But this idea's big (i.e. it solves a lot of GDPR issues, it potentially means that I no longer have to contact 50 different companies when I move house to tell them I've moved, companies can save a fortune wasted trying to track down those who've not given them this information, etc.).

As such, it seems unreasonable to have one system which holds & controls all of this information. Security concerned users may want to run their own instance which holds their information; but still enable friends using the "status-quo implementation" to access their data. Alternatively, other companies may want to host their own flavours of such a system, so it's not a monopoly; but again we'd want users of any implementation of this solution to be able to see other users even if they're on a different implementation (in the same way you can use any email provider and still talk to friends on other mail providers).

Apologies that this is quite a loose question; I've tried searching but don't know enough terminology in this area to know the correct search terms, as all systems I've worked on to date have been centralised, or at most had a master database with multiple slaves, rather than being fully decentralised / federated.

I've come up with a few ideas on this for this:

• Having a centralised register, but leaving everything else decentralised. So any system providing this functionality adds their URI to the register after which all systems can see it... However I don't like this solution as if possible I don't want any centralised dependency.

• Have each implementation holding a list of approved providers (i.e. for any big companies hosting public solutions), and allowing users to add "ad-hoc providers" to their personal "trusted provider" lists to cater for special cases (e.g. friends running their own personal instances / companies running in house instances). Essentially a similar pattern to how SSL certificates are currently handled in browsers; i.e. the recognised CAs are provided by your browser's vendor (some of these lists being published for general usage), but you can always add your own CAs to your browser as the need arises.

Answer 1

Let's break down the problem:

• You need some way to identify users
• Given an identifier, you need to have some way to determine how to push/pull information

Additionally, you obviously need some kind of standard protocol and a way to manage access rights. Also, the user id should be something people can remember, not some IPv6 address or the like.

The E-Mail Approach

In the end, you don't get around some centralization. But you may be able to delegate this part to already established infrastructure.

Example: E-Mail, which has similar problems, solves both of these issues by including the host information (mail server) in the user identifier (address). While e-mail itself is quite distributed, it relies on DNS to resolve host names, which is - at it's root - centralized.^*

You can go a similar way. Either come up with your own naming convention (e.g. providerXY.com>>AlbertEinstein79) or just use the e-mail format. You can even go one step further and actually use e-mail for your system's communication, i.e. you don't define a protocol but an e-mail template that the end-points can parse - not that I would necessarily recommend this route.

One downside of this approach is that it ties the user id to the underlying host, which means you can't really move your account without creating a new identifier.^**

^{* The centralization comes from the fact that everyone recognizes a few servers as authoritative.}

^{** If your system is push-based, you could use it to notify subscribers about your new account. However, this doesn't work if someone has written down your id on paper and tries to access it after your move. Anyways, push based is probably not the way to go.}

Lookup Approach

If you want to decouple the host from the user identifier (so that you can switch hosts), you need some kind of lookup service. If this should be decentralized, there are some problems to be solved.

• Ensure uniqueness of IDs
• Ensure that only the owner of the ID can change the host
• Ensure that one person cannot just create a couple billion IDs

While I'm not sure about (but open to) storing the actual data in a blockchain, it would actually make a good lookup store. I'm not really an expert on blockchains, so I'll refrain from considering the details here, but it seems to me the above mentioned problems might be solvable this way.

Answer 2

First @VladislavRastrusny is right.

This is not a pattern problem, a pattern is a general abstract way of solving an abstract known problem, that you then (try to) apply to a concrete case.

Here @JohnLBevan you seems more to have difficulties transforming an app/service/platform idea to "reality".

So to follow my comment and elaborate:

Blockchain at is core is a decentralized, distributed and hard to temper(secure to some good enough extend) database system.

If every user as his own block with his informations, and the "app" let him(or other service/app on his phone/computer automatically) search for other people information.

The "link" or "transaction" part can be used to give access to information, so in our situation if Bob authorize Alice to have his informations we have transaction et Bob and Alice are linked.

Blockchain as a system call PoW(Prove of Work) and can guarantee that only Bob is able to alter his information, therefore everybody in the network can be pretty confident that they'll not be another person pretending to be Bob.

Bob is only responsible for his entry and the validity of it. When Alice wants to, for example, call Bob, the app search for the Bob block and if it is considered valid(through PoW) by the system, transfer the phone number to the dialing app, and voila.

Of course this only works if Bob has an up-to-date entry but this problem is already here, and by having to maintain only your entry, the cost is fairly low, that's the all point right.

This is why your idea is great, because your app can become a decentralized, distributed IAM system(you'll need to secure the access to users devices with MFA).

Note that there is an alternative to Blockchain called Hashgraph you may want to look into that too.

Anyway, it will not be easy by any mean, so you'll need to surround yourself with the best people you can.

But again it seems like a great and ambitious idea, so I wish you good luck.