We have a database backing an iOS and Android app whose primary data is a simple text field. We have a build tool which builds the database - taking a raw txt
file which is a custom data format, which looks like:
---
.id SOME_ID
.author AUTHOR_NAME
.field MORE_DATA
.data_title TITLE
Data data data [#tag, #tag, ...]: some more data
Field: [id] description of data; [id] description
Examples:
- e.x. some example describing the entry
- e.x. another example
Closing field: more data; more data
---
or something along those lines.
To build the database, a parser was implemented to take the raw txt
file, apply some transformations/optimizations to save storage space (e.g. compressing similar entries), and then convert it to a binary format. The resulting entry is stored in a similar but compressed format in the database, also as a raw text field that we'll call entry_data
in table entries
.
All consumers of the database, because the data is stored essentially in the same format, also need to implement a very similar parser, but for the compressed format in entry_data
. That means Android and iOS both have about 20-30 files handling the parsing, as well as the web app that employees use to update the database.
I believe that the initial reason for this custom format was due to ease-of-adding for the people working on the database, as well as for storage space reasons (I'm not sure, but I think 6-8 years ago mobile devices had low limits on available storage). In any case, the database is only about 8 MB compressed, but is split into several chunks, also probably to avoid file size limit restrictions on very old devices, which we no longer even support.
The biggest issue with this custom format is that even small changes, such as adding fields, causes a cascade of changes throughout the web tool the people working on the database use, the database build tool, as well as all clients. So if we want to add a category
field, we need to make changes to parsers in 4 different code bases, all of which are in different languages (JavaScript, Java and Objective-C).
All of our data relationships can be modeled with a relational database. There's no reason to have examples embedded in the entry - examples could easily be linked via an entry_examples
join table. There are several other fields which have duplicate data across entries, which could also be solved by simple foreign keys.
My other thought is that data storage space is much less of an issue in modern devices - if losing the complex compression inflates our database from 8 MB to 32 MB, which is what many other apps in the same category as ours have (or even a lot more), I don't think users will mind, especially if changing to a standard data format allows us to add features in the apps more quickly.
Does it make sense to have custom data formats like ours, when there are widely-adopted standard formats for data (e.g. relational structure in relational databases, or even NoSQL databases that have the entries as structured JSON)?