2

I am in the process of building an heavily data-driven web application written in Node.js / Typescript with Mariadb for the database engine. I am trying to put an emphasis on extensibility and DRY principles.

One thing that I've come across that is repetitive and time consuming to write code for is data validation.

Some background on the application:

  1. There is a base Querynator class used to provide basic convenience methods such as getById and createUpdate
  2. Much of the schema was inspired by the ServiceNow platform such as using uuid for primary keys on every table and general nomenclature.
  3. API calls are completely table based. e.g. /api/table/sys_user or /api/table/sys_user/94a993ab-4187-4791-b48f-3d79d5182df0
  4. The database schema is stored in a table separate from the INFORMATION_SCHEMA and is stored in memory when the application starts.

Currently I have 3 ideas on how to tackle this:

1. Custom classes for every table

With this method, each table will have it's own resolver which may or may implement a shared schema (see #2). This is how I imagine most applications would handle queries / updates etc.

Pros:

  1. Extremely granular control, easy to optimize individual tables.
  2. Very easy to implement custom logic on a per table basis
  3. Great if there are not that many tables

Cons:

  1. Tedious to code for applications that may have over 100 tables
  2. More difficult to maintain
  3. Higher likelihood of 1 change breaking many things.

2. Schema based validation

With this method, all data is validated against the schema, which I maintain in a table separate from the INFORMATION_SCHEMA. When modifications are made to the schema, data is automatically validated on HTTP requests. In the case of GET requests, the only thing that is validated are the fields provided in the fields and args query string parameters.

Pros:

  1. Only valid data is provided to the database
  2. Extensible so long as the table information is maintained in the schema
  3. Can be extended to the client to some degree

Cons:

  1. Custom logic needs to be implemented for special fields, such as the password and verify password on the user registration form.
  2. If custom logic is implemented, it must be included in the source code

3. Hook based validation

This validation method would also validate data against the schema, but it would also provide another programming interface using "hooks", similar to git hooks. Hooks are simply events that are fired in the request lifecycle. Hook handlers would be stored in a table that looks something like this:

CREATE TABLE sys_db_hook (
    PRIMARY KEY(sys_id),
    sys_id CHAR(36),
    table VARCHAR(40),
    hook VARCHAR(20),
    code TEXT,

    UNIQUE(table, hook)
)

A sample record would look something like this:

{
table: `sys_user`,
hook: `onBeforeUpdate`,
code: `var Querynator = require('querynator')

module.exports = function(id, newFields) {
    this.status = {...}
    // Do stuff

    return this.status
}`

My implementation would look something like this:

  1. Before listener is started, server reads data from sys_db_hook table and exports the code to a hooks directory with filenames in format ${hook}_${table}.js
  2. Client makes a request to server update to any table.
  3. Internally, the Querynator object emits a hook event.
  4. Hook handler looks for existence of file with the ${hook}_${table}.js in the hooks directory.
  5. If the hook is found, then the javascript is dynamically required.
  6. The hook executes and returns the mutated object

Of course the hook would be restricted with what it could access. Before the hook is updated in the database, there is a regexp test for require(), if any matches are found besides querynator, then that line is commented out. Same goes for global vars like process (replace with _process) and __dirname etc...

Pros:

  1. Incredibly extensible. Anything could be added to the application and a hook could be created to handle mutations, etc...
  2. No source code modifications necessary.
  3. Querynator can be coded in such a way to provide many useful functions for manipulating data or integrating with external resources.
  4. Potential for reusing code on the client as well as the server

Cons:

  1. Considerable performance consequences depending on how the script is written
  2. Possible memory issues relating to loading dozens or hundreds of extra javascript modules.
  3. Testing would be very difficult to implement other than checking for the existence of errors and execution time.

My Question Is: Obviously the hook is a much more flexible utility, but I am concerned with possible performance issues related to using it. Also, if something like the hook method above were used, would it make more sense to use a message broker rather than execute on the main thread (even if cluster were used)? Or does a message broker not make sense in this context?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.