Does it make sense to have a custom textual data format?

Question

We have a database backing an iOS and Android app whose primary data is a simple text field. We have a build tool which builds the database - taking a raw txt file which is a custom data format, which looks like:

---
.id SOME_ID
.author AUTHOR_NAME
.field MORE_DATA
.data_title TITLE

Data data data [#tag, #tag, ...]: some more data

Field: [id] description of data; [id] description

Examples:
 - e.x. some example describing the entry
 - e.x. another example

Closing field: more data; more data
---

or something along those lines.

To build the database, a parser was implemented to take the raw txt file, apply some transformations/optimizations to save storage space (e.g. compressing similar entries), and then convert it to a binary format. The resulting entry is stored in a similar but compressed format in the database, also as a raw text field that we'll call entry_data in table entries.

All consumers of the database, because the data is stored essentially in the same format, also need to implement a very similar parser, but for the compressed format in entry_data. That means Android and iOS both have about 20-30 files handling the parsing, as well as the web app that employees use to update the database.

I believe that the initial reason for this custom format was due to ease-of-adding for the people working on the database, as well as for storage space reasons (I'm not sure, but I think 6-8 years ago mobile devices had low limits on available storage). In any case, the database is only about 8 MB compressed, but is split into several chunks, also probably to avoid file size limit restrictions on very old devices, which we no longer even support.

The biggest issue with this custom format is that even small changes, such as adding fields, causes a cascade of changes throughout the web tool the people working on the database use, the database build tool, as well as all clients. So if we want to add a category field, we need to make changes to parsers in 4 different code bases, all of which are in different languages (JavaScript, Java and Objective-C).

All of our data relationships can be modeled with a relational database. There's no reason to have examples embedded in the entry - examples could easily be linked via an entry_examples join table. There are several other fields which have duplicate data across entries, which could also be solved by simple foreign keys.

My other thought is that data storage space is much less of an issue in modern devices - if losing the complex compression inflates our database from 8 MB to 32 MB, which is what many other apps in the same category as ours have (or even a lot more), I don't think users will mind, especially if changing to a standard data format allows us to add features in the apps more quickly.

Does it make sense to have custom data formats like ours, when there are widely-adopted standard formats for data (e.g. relational structure in relational databases, or even NoSQL databases that have the entries as structured JSON)?

Answer 1

Does it make sense to have custom data formats like ours, when there are widely-adopted standard formats for data (e.g. relational structure in relational databases, or even NoSQL databases that have the entries as structured JSON)?

No.

The thing is, not only do things like JSON and relational databases exist, they're long-time standards, which means strong, robust ecosystems of tooling has been built around them. All the common things you might need to do with your data, someone has already published an open-source library to take care of, which you can get access to for free. If you start from scratch with a proprietary format, you have to build all that tooling yourself. You're going to make mistakes along the way--mistakes that the people who wrote the libraries for standard formats have already made and fixed.

This is exactly the sort of scenario that the phrase "don't reinvent the wheel" was created for. If the data is small and simple, use JSON. (Not JSON-backed NoSQL databases; just flat JSON text files.) If it's a large dataset and you need to run non-trivial queries against it, use a relational database.

Also, a bit of free advice: Don't use NoSQL databases unless you're actually running something "web-scale" (ie. on the same order of magnitude as Amazon, Google or Facebook) as the hassles it brings tends to outweigh the benefits until you're big enough that those benefits are actually necessary rather than "nice to have." The ACID guarantees the relational model offers can be the best thing that ever happened to your data, if you let them.