Golang Design Pattern for Generating View Objects in a REST API?

Question

I have moderate Golang experience and lots of experience in other programming languages such as Java, Python, Rust, Scala, and others. I'm comfortable with building REST services and most of the other related topics, so the concept in general isn't new to me.

Background

I'm building a REST API in Golang using gorilla/mux as an HTTP router, gorm as an object-relational mapper to the database (which is MySQL 5.7), swaggo/swag to auto-generate Swagger endpoints and models from code comments, and a few other libraries for JWT and other related boilerplate.

I have repeatedly run into circular import problems, they seem to be very easy to trigger, reminds me of Python but seems more easy to trigger, and so I've setup my codebase in a way where the only thing that is touching database models and Swagger models are the actual HTTP handlers themselves.

Directory Structure

My directory structure looks somewhat like this:

main.go
pkg/models/*.go (HTTP JSON models)
pkg/models/db/*.go (Gorm Database Models)
pkg/routes/*.go (HTTP Handlers)

HTTP Route Package Structure

I have a sort of nested routes structure, which has so far worked out for me. Every subpackage in routes has a function to configure the routes:

func ConfigureRoutes(r *mux.Router) {
    r.Path("/create").HandlerFunc(CreateEventHandler).Methods("POST")
    // register subpaths
    bookings.ConfigureRoutes(r)
}

I have written middleware and a lot of little helper functions to make my life easier.

Generic Response

All JSON models are contained in a generic struct:

// Response The default wrapper response model used in all responses.
type Response struct {
    // The HTTP status code of the response.
    Status uint16 `json:"status" example:"200"`
    // The error message, if any, associated with the response.
    Message string `json:"message,omitempty" example:"Error message."`
    // Data returned by the request, if any.
    Data interface{} `json:"data,omitempty" example:""`
}

Database Models

Here is a database model in gorm, existing at pkg/models/db/events.go:

// Event An event booked at the hotel.
type Event struct {
    Type         uint8
    Name         string
    Description  string
    Notes        string
    Start        null.Time
    End          null.Time
    MaxPerPerson uint32
    Capacity     uint32
    Price        decimal.Decimal // FIXME need something that uses MySQL DECIMAL type: https://dev.mysql.com/doc/refman/5.7/en/precision-math-decimal-characteristics.html
    gorm.Model
}

HTTP Handlers

I have been able to build a lot of the API so far by just copying struct fields in HTTP handlers like so:

// EventCreateHandler godoc
// @Summary Create an event.
// @Accept json
// @Param event body models.AdminNewEventRequest true "New event data."
// @Produce json
// @Success 200 {object} models.Response{data=models.AdminNewEventResponse}
// @Failure 403 {object} models.Response
// @Security JWT
// @Router /admin/events/create [post]
func EventCreateHandler(w http.ResponseWriter, r *http.Request) {
    if !request.IsAdmin(r) {
        response.Unauthorized(w)
        return
    }

    var reqBody models.AdminNewEventRequest

    if err := request.JSONBody(r, &reqBody); err != nil {
        response.BadRequest(w)
        return
    }

    var event db.Event

    event.Name = reqBody.Name
    event.Description = reqBody.Description
    event.Notes = reqBody.Notes
    event.Start = null.TimeFrom(time.Time(reqBody.Start))
    event.End = null.TimeFrom(time.Time(reqBody.End))
    event.MaxPerPerson = reqBody.MaxPerPerson
    event.Capacity = reqBody.Capacity

    price, err := decimal.NewFromString(reqBody.Price)

    if err != nil {
        response.BadRequest(w, "Unable to parse price as decimal.")
        return
    }

    event.Price = price
    event.Type = reqBody.Type

    conn := request.GetDB(r)
    tx := conn.Save(&event)

    if tx.RowsAffected == 0 {
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    response.Ok(w, models.AdminNewEventResponse{
        ID:           event.ID,
        Name:         event.Name,
        Description:  event.Description,
        Notes:        event.Notes,
        Start:        models.JSONDateTime(event.Start.ValueOrZero()),
        End:          models.JSONDateTime(event.End.ValueOrZero()),
        MaxPerPerson: event.MaxPerPerson,
        Capacity:     event.Capacity,
        Price:        event.Price.String(),
        Type:         event.Type,
        CreatedAt:    models.JSONDateTime(event.CreatedAt),
        UpdatedAt:    models.JSONDateTime(event.UpdatedAt),
        DeletedAt:    models.JSONDateTime(event.DeletedAt.Time),
    })
}

JSON Request/Response Models

For this particular data-type, I have many structs:

• models.AdminNewEventRequest: Contains all the fields except for ID and contains some fields which are optional, and using gopkg.in/guregu/null.v4 for those fields which can be omitted.
• models.AdminNewEventResponse: Contains all the fields including the ID and the auto-generated database timestamps (CreatedAt, UpdatedAt, etc.)
• models.AdminUpdateEventRequest: Contains all the fields except for ID and CreatedAt/UpdatedAt, but every field definition is optional, as this is a PATCH request, so each field is something like Name null.String etc.
• models.AdminUpdateEventResponse: Similar to AdminNewEventResponse.
• models.AdminGetEventResponse: Similar to AdminNewEventResponse.
• models.AdminListEventsResponse: A paginated container for a list of events.

---

Problem

• I have about 12 different data-types to model, so 12 different gorm models.
• Each data-type has about four different HTTP handlers, so that means about 48 handlers at minimum, some have more than four.
• Each HTTP handler has at least one JSON model and up to two of them:
  1. For GET/DELETE handlers, I only have a response model, like models.AdminGetEventResponse
  2. For POST/PATCH handlers, I have both a request and a response model, like models.AdminUpdateEventRequest and models.AdminUpdateEventResponse
• The net result is that I have literally 88 JSON models.

Eighty-eight JSON models.

And because I fear triggering the circular import problem, in every handler, I'm copying every single field, line by line.

Updates are worse: I load the model from the database, and since every field in an update request is optional, I have to do this monstrosity:

var reqBody models.AdminUpdateEventRequest
// parse the json model from the request body
var dbModel db.Event
// load from the database

// oh lord here we go
if !reqBody.Name.IsZero() {
    dbModel.Name = reqBody.Name.ValueOrZero()
}

// repeat for EVERY SINGLE FIELD

Question

Since I'm giving disparate views on every database model, I can't just de/serialize the database models to/from the request/responses; for example, a PATCH request for updating a user includes an optional Password field which will cause the KDF to generate a new salt/KDF config/hashed password and store it in the database, and this does not 1:1 match the database model, especially because every field is optional (null.String, etc.). Since every JSON request/response model is not directly tied to the database model, I have to define every request/response model by hand, and in every HTTP handler, I'm literally just copying fields around everywhere and some of my database models have 25 fields.

Having 88 models for just the admin views on the 12 database models feels extremely excessive, and (optionally) copying every single field for these models for request to database model and database model to response is getting extremely tedious.

I might be able to reorganize things so that JSON models can refer to database models in part without circular imports, but I haven't done that yet because it'd be non-trivial refactor.

I feel like I'm doing something terribly wrong here, but I can't see another way to provide custom views on my database models to satisfy my API requirements. Perhaps I need to dive into reflection (*massive disappointment face*) or some kind of code generation system, but since this is my first big REST project in Golang, I don't know where to look or how these things are typically done.

• 12 data types.
• 88 admin request/response models.
• 58 HTTP handlers for various tasks.

It will only get worse as I add non-admin routes and JSON models.

Is there a better, faster way to be doing this?

Answer 1

Very interesting technologies you are using. I have debated the same thing in a different implementation with Spring Boot services. My choice was to separate the database entities from the models being returned to the client. It seems like there needs to be some intentional layer of separation there since if someone inadvertently added fields to the database and the entities it could accidentally break a service contract. In Java there are helpers that AutoMap between entities and data transfer objects which might help but like you I have a factory that builds these mappings manually. I would be interested to see what others have to say here as well.

A quick Google lookup shows that there might be automapping options for Go as well, so might want to give that a try.

For example: https://pkg.go.dev/github.com/PeteProgrammer/go-automapper

Final point I wanted to make is that from experience I have noticed that it is easy to get carried away and expose an API for each model when in fact many models are often bundled as part of a parent model and might not deserve their own endpoint.

Answer 2

As per a prompt from another answer, I searched for and found some auto-mapper libraries and tried them out. The general idea is that reflection is used at runtime to map similarly named fields across disparate structs.

I tried the following:

• github.com/stroiman/go-automapper
• github.com/devfeel/mapper
• github.com/ulule/deepcopier

automapper and mapper lacked certain features around being able to rename/exclude certain fields, but deepcopier works for 2/3 of my use-cases.

Here are test cases showing it working:

package coherence

import (
    "github.com/stretchr/testify/assert"
    "github.com/ulule/deepcopier"
    "gopkg.in/guregu/null.v4"
    "gorm.io/gorm"
    "testing"
    "time"
)

type DatabaseModel struct {
    Name           string
    Description    string
    NullableString null.String
    Timestamp      null.Time `deepcopier:"skip"`
    Password       string    `deepcopier:"skip"`
    gorm.Model
}

type CreateRequestModel struct {
    Name        string
    Description string
}

type UpdateRequestModel struct {
    Name        null.String `deepcopier:"force"`
    Description null.String `deepcopier:"force"`
}

type ResponseModel struct {
    ID             uint
    Name           string
    Description    string
    NullableString null.String
    Timestamp      models.JSONDateTime
}

func TestDeepCopierCoherence_Response(t *testing.T) {
    dbModel := DatabaseModel{
        Name:           "Hello",
        Description:    "World",
        NullableString: null.StringFrom("x"),
        Timestamp:      null.TimeFrom(time.Now().UTC()),
        Model: gorm.Model{
            ID: 12345,
        },
    }

    httpModel := ResponseModel{}

    if err := deepcopier.Copy(&dbModel).To(&httpModel); err != nil {
        t.Fatalf("Unable to deep copy: %s", err)
    }

    assert.Equal(t, dbModel.ID, httpModel.ID)
    assert.Equal(t, dbModel.Name, httpModel.Name)
    assert.Equal(t, dbModel.NullableString, httpModel.NullableString)
    assert.True(t, httpModel.Timestamp.IsZero())
}

func TestDeepCopierCoherence_CreateRequest(t *testing.T) {
    dbModel := DatabaseModel{}

    httpModel := CreateRequestModel{
        Name:        "Name",
        Description: "Description",
    }

    if err := deepcopier.Copy(&httpModel).To(&dbModel); err != nil {
        t.Fatalf("Unable to deep copy: %s", err)
    }

    assert.Equal(t, httpModel.Name, dbModel.Name)
    assert.Equal(t, httpModel.Description, dbModel.Description)
}

However, the use-case that is not covered is updating existing values. PATCH requests are defined as UpdateRequestModel is above: every field is optional utilizing null.* types, but deepcopier is not aware of this package, so unless you are copying 1:1 (e.g. null.String to null.String), this approach does not work for PATCH requests with all fields optional.

I may PR the codebase to add support for the null package, but even this will save me a tremendous amount of time. If anyone knows of a workaround or a library which will respect null.* types, that would complete basically all of the use-cases that I have.