How to handle relationships between resources in Restful API

Question

Problem

Hi, we have an API that it's pretty similar to a standard CRM. That is, we have a really big amount of resources (even our own customers can define new resources). All of these resources share the same endpoint: /resources/<resource_name>. This resource name can be any name, like customers, leads, contacts, etc.

Every resource is different from one another, they only share some common fields like the resource owner, last modified date, and so on.

Now, as is usual in CRMs, we want to implement relationships between those resources. As the own users can define their resources and their relationships, we need an API that can handle just that: 1.1 relationships, 1.N, N.N, and so on.

Proposed solution

Include in our resource response a relationship field, containing the information needed to implement this new feature. This is what a normal resource response looks like in our API:

{
  "id": 123,
   ... more common fields to all resources ...
   "fields": { ... fields specific to this resource ... }
}

And this is what it would like after the relationship field is implemented:

{
  "id": 123,
   ... more common fields to all resources ...
   "fields": { ... fields specific to this resource ... }
   "relationship": {
       "a_related_resource_name": [list of resource ids],
       "another_related_resource_name": [list of resource ids]
   }
}

Use Case

Our current solution covers all use cases of the API:

1. Relate a resource instance to another: given a lead resource with id 283, relate it with a customer resource with id 284. That would be just doing a PATCH request to /resources/lead/283

{
   "relationship": {
       "customers": [284],
   }
}

2. Edit the relationship of existing resources (including deleting the relationship): just modify the content of the relationship key to reflect what state we want. If we want to delete all the relationships between our lead resource and its customers, then we just set customers to an empty array. If we want to relate our lead to another customer, then we just add the id of that customer to the array.

Solution drawbacks

I think this solution works nicely if the number of resources in a relationship is not too big. That is, if N is small in the 1.N and N.N relations. If N is too big, then the bodies of our requests and responses are going to start growing too much, making our clients too slow.

Question

Now that you have a hold of the problem, what do you think about the solution? Any idea how to make it scale when N is big? I'm pretty sure this problem has already been solved by some CRM or some API design pattern, but I have run out of ideas about how to design this feature efficiently and with an API easy to use.

If you need any clarification please just tell me so!