1

Let's say I have a users resource, with two properties: name and email as specified by a users JSON Schema document, which right now looks like this:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "name": {
      "type": "string"
    },
    "email": {
      "type": "string"
    }
  },
  "required": [
    "name",
    "email"
  ]
}

My requirements state that we need to be able to change the schema, e.g. to add a property such as phoneNumber, and do so via HTTP in a RESTful way. That is, I need to be able to update the JSON Schema definition of the users resource to look like this:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "name": {
      "type": "string"
    },
    "email": {
      "type": "string"
    },
    "phoneNumber": {
      "type": "string"
    }
  },
  "required": [
    "name",
    "email",
    "phoneNumber"
  ]
}

Now, clients of the API can create new users that have the additional phoneNumber property (where previously I would have gotten a schema validation error).

I am puzzling over how to do this. One way I can imagine doing it is by creating a "meta-resource" called resources. This resource might have some properties, for example: path and schema. The schema property would be a full JSON Schema object. To update the users resource, then, I could maybe POST to resources with an HTTP request body like:

{
    "path": "users",
    "schema": { ...JSON Schema object goes here... }
}

Is this a reasonable implementation? If not, why not? Alternative ideas? Any pitfalls I should watch out for? Any articles/blogs on this topic that I should read? (I haven't been able to Google successfully for this).

  • 1
    related question : softwareengineering.stackexchange.com/questions/338102/… – linuxunil Jan 3 '17 at 18:46
  • Can you clarify why you need a schema validation in the first place? Are the users of the system who update the schemas of each resource different to the users who upload the resources themselves? In other words do you still want/need to validate the resources when they are uploaded even though the schema might be changing every few months? If not then it seems unnecessary to have a schema validation on the server in the first place, you could just trust that the uploaded resources are in the schema the end user wants. – Cormac Mulhall Jan 4 '17 at 18:06
  • Think of it as a set of assertions in some code. Even if I am the only developer and user of the code, I still want to ensure that I don't make a mistake and provide invalid arguments. This is partly for that reason, and also partly because the analysts who will use this API to load data are relying on it for part of their quality control--we don't want to allow strings where we expect integers, and so on. – user92338 Jan 4 '17 at 23:04
2

The simplest solution here would seem to but to call PUT on the user resource with the new version of the schema. This is a problematic in general because it's subject to dirty reads. This is also unworkable for your specific scenario due to the versioning requirements that you have noted in the comments below.

As I understand it, a given version of each schema is immutable*. Given that requirement, I think best answer here is that when a change is made to the schema, you would do a POST to the resource root and let it determine the new version number. For example, say you have resource /schema/user/1, if you want to modify it, the schema is retrieved, edited and POSTed to /schema/user which will create resource /schema/user/2. You could try to do some sort of PATCH action against the schema and let it create the new merged version for you but that is far more difficult to implement and I don't think it would be worth it unless there is some specific need for it to happen on the server.

*You may need to make accommodations as necessary to support corrections.

  • I'm afraid that this response misunderstands my question. I have since updated my question to clarify. This answer seems to think that I am asking how to update a user resource, e.g. users/123 is missing a phoneNumber and I need to add one. That's not what I'm asking. I'm asking how to update the users definition, e.g. the JSON Schema that defines valid content for the users resource. I want to allow users to have a phoneNumber property in the first place. – user92338 Jan 3 '17 at 21:53
  • You are right, I did not put that together from your original question. Much of this still stands but I will adjust as needed. One thing that is not clear is how you will handle documents created under prior versions of the schema. For example, if a new required element is added to the schema, all existing documents would be invalid. Are you planning to version the schemas? – JimmyJames Jan 4 '17 at 15:34
  • You put your finger on an important question in our implementation. Currently, our plan is to serve the resource as it was written. Each schema will be versioned, and each resource will be given a schemaVersion property to indicate which version of the schema was used to validate it when it was written. New resources will always be created after being validated using the latest version of the relevant schema. – user92338 Jan 4 '17 at 23:01
  • I have edited my answer based on your feedback. – JimmyJames Jan 6 '17 at 15:05
0

If you really want to modify the data schema from an external source, you'll probably want to ensure that there is a good authorisation system in place to ensure that schema updates are happening in a permitted way. Can you imagine what would happen if someone managed to send malicious schema-altering messages that resulted in all attributes being dropped from everything? I'd also recommend tracking every change that comes in.

In general:

  • why update the data schema remotely? Does your domain really change that much that it's necessary to be able to simply add/remove attributes to types in the schema?
  • Have you considered how clients are supposed to respond to this changing schema? Do they expect frequent changes?
  • What's the underlying datastore? An SQL database? A key-value store? Will existing data need to be changed every time you changes the schema? Will underlying data structures (such as SQL databases) need to be ALTERed every time this happens?
  • What about any generated code that matches the schema, will all that need to be regenerated too?

I've never seen this done before. Is there a compelling reason to be able to do this?


But if you really want to do this...

This operation is different enough that I think it should have its own "schema-update" endpoint. Maybe something of the form:

http://myserver.com/schemaManagement/(maybe optional schema version number?)/update/$objectType/$operation/$newAttribute

So adding the phoneNumber to person might involve a POST to http://myserver.com/schemaManagement/current/update/person/add/phoneNumber and the content of the message could include details about the datatype being added, whether or not is is optional (and if not, what default value to use), authorisation tokens, etc...

Removing an attribute might look like:

http://myserver.com/schemaManagement/current/update/person/remove/faxNumber

I'm not a REST expert but this seems to me to be a pretty intuitive and easy-to-manage way to handle something like this.

  • I totally agree re: authorization and change tracking. We have those things nailed down. In order: 1. Why? We are building a system for analysts to create, store and update arbitrary datasets. These have a structure that changes from year to year. New ones are added and the analysts want to be able to add and change them. 2. The clients are the same analysts (this is an in-house system), using ETL tools; they expect the change. 3. Stored in a JSON document-based backend with complex schema-specific revisioning. 4. Yes, but this will be easier than our present approach. – user92338 Jan 3 '17 at 22:30
  • That said, I'm not saying this is the best idea, just that it is presently a requirement. – user92338 Jan 3 '17 at 22:31
  • @Stew: I've added an update that I think might help. – FrustratedWithFormsDesigner Jan 3 '17 at 22:38
0

If there is a possibility to change a schema remotely, then there must also be a mechanism for clients to retrieve the current schema.
If you combine that with a REST interface, then it becomes fairly logical to make the schema itself also a REST resource.

This would mean that you would retrieve the current schema for the user resource with

GET http://example.com/api/schemas/user

and you change the schema with

PUT http://example.com/api/schemas/user

and other resources with a schema similarly. The resource targeted by the schema becomes the key into the schemas collection.

0

How do you handle the updated schema in your Web service?

Using the approach in your question, you probably will have to update your backend code as well. So I do not think that changing/updating resource definitions via RESTful API is a good idea.

If the backend is able to handle dynamic data, I would suggest to use a dynamic schema. A simple example could be a key/value list.

The JSON schema could be something like that:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "values": {
       "items": [{
        "properties": {
         "key": {
          "type": "string"
         },
         "value": {
          "type": "string"
         }
        },
        "type": "object"
       }],
       "type": "array"
      }
  },
  "required": [
    "values"
  ]
}

If your backend changes frequently, it could make sense adding an endpoint to GET an updated schema definition from the backend.

0

Well, you can try to accept JSONPatch documents when you do PATCH HTTP request.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy