I apologize in advance for the length of this question; it requires some explanation. I'll try to make it as clear as possible.
I am developing and maintaining a mobile app (Xamarin, F#) that communicates with a web API (ASP.NET Core, F#). The server-side API exists only to communicate with the app and do database operations and other HTTP requests as needed. The domain model, so to speak, is more or less exactly the same in the app and the API. For example, both the app and the API have a concept of a
CardId with the same notions of what exactly constitutes a valid card ID.
Since apps can't be instantly upgraded, I must version the API so that users of outdated app versions can still use the service. Currently the app and server communicates by serializing/deserializing simple DTO-like types (sent as JSON payloads over HTTP) defined in a common assembly referenced by both the app and API. For example, a card ID is represented as a simple string. The types in this shared assembly are versioned, and the app and API convert between these types and their corresponding types that they use internally.
On the app side, the app converts between internal representations (defined in the app assemblies) and the latest version of these types (since the app only ever needs to work with a single API version).
On the API side it's similar, but the API contains converters for all versions of the shared types, since it needs to support older versions too.
This all means that any concept/type shared between the app and server needs at least 3 types (for a single version) - one in the app, one in the API, and one in the shared assembly to be serialized/deserialized - as well as 2-4 converters (1-2 each in the app/API, depending on whether it needs to only receive/send the types or both). This is very robust as it decouples the internal representation (and the logic working on them) from the communication between app and server, but it's a heck of a lot of boilerplate. Also, it leads to me often cutting logic from the app - for example, a card ID in the app is just modeled as a simple string, and the API does validation on it and can return an error code indicating "invalid card ID".
Now, what I really want to do is to simply share objects between client and server. If they could somehow share memory (absurd, but explains my point), that would simplify things a lot. Since both the client and server use the same platform/language (.NET/F#), and this is unlikely to ever change (if so, one could always go back to the current solution), I have considered collapsing all three versions of a type (app, API, and shared) into a single type (still versioned, of course), and simply serialize/deserialize that. For example using some kind of binary serialization (I've no experience with that).
This ensures that the server and app are truly speaking the same language and that equal concepts are treated equally. For example, I can share complex objects with guarantees on some invariants by construction (such as the abovementioned CardID) and know that all invariant still holds once I deserialize it on the other end, because it's the exact same object. I could then also get rid of several server-side checks and errors like "Invalid card ID" because the app could never construct and send an invalid card ID in the first place.
It also means that all across my app and API, there would be references to a specific version of objects, but I don't think that would be any worse to maintain than the current solution.
However, I can't shake the feeling that I'd somehow be shooting myself in the foot, or that versioning would be more difficult and brittle, though I can't pinpoint why I feel that way (perhaps it just feels more robust with dedicated objects that I use for serialization/deserialization and convert to/from internally, but I'm still relying on serialization/deserialization of these DTO objects, so why not go all the way?). The question is therefore:
In light of the above explanation, is sharing objects between client and server using e.g. binary serialization a good option, or have I missed some clear problems with that approach? Have I perhaps missed another better option entirely?