8

Context: I'm an embedded dev with only 2 years of solid experience. I'm the sole technical employee of a startup of 4 people. We have an MVP of our product out and are getting ready to develop the next iteration of it. The original MVP was developed by a partnered contractor team, with one older embedded dev doing everything software. I joined the company too late to have any input into the design of the MVP. The product is a gateway-type device: embedded Linux, messages come in one way, some limited intelligence happens inside, they come out on the other side.

The 'problem': Everything in the system seems to be chucked into a single SQLite database. Processed and unprocessed messages live in the same table (with one field used to indicate which one they are), provisioning related things live another table, even the logging and debug is done by writing to yet another table in the database. The system is written largely in python but the biggest part of it is a massive class full of wrappers for complex SQL which seem to do the bulk of the data manipulation.

All of this makes me uncomfortable, especially the messages part as it looks like a classic case of "database as a queue" design anti-pattern. That being said I find it difficult to articulate exactly why this is wrong to my non-dev boss, other than vague and nebulous mentions of maintainability, difficulty in introducing changes because of lack of modularisation, as well as of lack of clarity in how data flows and is being processed. It doesn't help that the contractor has "authority of years of experience" over my judgment.

Am I justified in feeling uncomfortable about this design choice? I mean, the thing works in principle. I know that the desire to refactor can be pretty strong, irrational and should not always be acted upon given business constraints. But then... I kinda feel like starting mostly from scratch only ripping out useful bits of the first MVP would be cleaner and take less time than working with the system created so far. But is it just my younger-person-hot-headedness?

  • 12
    wait, why is database as a queue an anti-pattern? I can see where the atomicity and other guarantees provide a lot of benefit, particularly in this sort of scenario where everything is on-device. – Telastyn Jan 27 at 17:14
  • 3
    Keeping the messages in a single table and flagging their status seems like a good idea to me. This way you're not needlessly moving them around. – GrandmasterB Jan 27 at 17:37
  • 3
    Have you considered the case of unexpected frequent power loss and resets? This is what embedded systems have to cope with frequently. One approach to this, which has found popularity among mobile app developers as well, is to persist often: in a SQLite database. – rwong Jan 27 at 17:55
  • 2
    An embedded device using SQlite? Sounds fine to me. Remember, SQLite isn't a replacement for RDBMS systems like MySQL, it's a replacement for fopen(). sqlite.org/whentouse.html – whatsisname Jan 27 at 18:24
  • 2
    Added: before the "new persistence approach" (SQLite) became popular, embedded systems were: often not using any OS at all; coded "on the metal", in assembly or C (no C++), uses hardware-specific compiler modifications and built-in libraries, have direct software access to RAM and EEPROM (and the ability to rewrite the latter), have no file systems whatsoever. The fact that "Python" is used implies that it is not an "old" embedded system. – rwong Jan 27 at 19:15
22

Your first problem is that you think of the design as "wrong." That's really not the right way to consider things. Rather, different designs make different design trade-offs. Any design has pros and cons that have weighed against each other. If there is a problem with the design it's not that is "wrong" but rather that it makes a poor choice of trade-offs. Don't think of it or explain it as, the design is wrong, rather think in terms of how a better trade-off might exist.

The fact that you are using SQLite and an embedded Linux system makes database as queue far less problematic then it would be in other circumstances. Database-as-queue has two major drawbacks:

  1. Database operations are fairly expensive, reducing the performance.
  2. New messages have to be detected by repeatedly querying the database to check for them.

However, SQLite operations are much less expensive than a typical database, so the performance loss is minimal. SQLite also supports a data changed notification that allows the program to detect changes to the database without repeatedly querying the database.

SQLite gives you a lot of functionality for free. Basically, you define your tables, and SQLite takes care of persisting them to disk, loading them back when you restart, ensuring that the file does not get corrupted, giving you powerful querying tools, allowing complex operations on the data. Furthermore, SQLite gives you all of this without configuring other applications, its just a library your code is using. If you don't use SQLite for all of this stuff, you'll have to implement that functionality yourself.

In my assessment, using SQLite in this way is a pretty sensible choice. Your response comes across to me as somebody who hasn't used databases enough to know their power. Of course, there may be other considerations I don't know that change that, or it could be the system is simply poorly constructed in other ways.

But if you do want to take your concerns to your boss:

  1. Make a good faith effort to work with the system as it stands first, that will make it more plausible when you make the cast. Also, you'll have a much better idea of what does and doesn't work. And maybe you'll learn to love SQLite.
  2. Be able to identify precise things your boss cares about: better performance, more features, etc. in making the case.
  • I really appreciate this response - it's really the reason why I posted the question - to find out about the justification for the contractor's choice and to be able to choose how we proceed given that. I knew that using SQLite was likely to drastically decrease the amount of work needed to just make things work - now I just need to make sure that its interface code is not written like it's not meant to be read by anyone ever. – KubaFYI Jan 27 at 20:34
  • 1
    @KubaFYI Speaking as a dev with a legacy codebase older than myself, "surgically" making changes is your best approach. Don't touch more than you need to but definitely document where you can. Eventually, by documentation alone the codebase will become more maintainable and easier to rewrite later. – IT Alex Jan 28 at 14:25
  • @KubaFYI, good luck! – Winston Ewert Jan 28 at 18:41
8

I think you are making a common junior-level programmer error in your judgement here. Because the solution is not the way you would have done it, you want to redesign it. One of the most important things you need to become comfortable with is that whether the design is the way you think it should be doesn't matter. It really, truly doesn't matter at all. All that matters is whether it works, whether it can be improved, and what what it will cost (time and money) to make those improvements.

You haven't mention a single concrete problem. You see a design you don't like. But why should your manager care? Have you produced any working production software solutions of a similar scale and complexity? They've got a perfectly good milking cow and you want to them to trade it in for some magic beans.

Let's start with the assumption that this software works. It's reliable and performs well. Customer satisfaction is high. Now you come in and tell the manager you want to rewrite it. Basically you are saying that the initial time and investment that went into building this is a sunk cost and that you want scrap a working, profitable piece of software for your unknown abilities.

Writing the code is just the beginning of getting to production. Most likely there are a lot of technical challenges that are solved in the current solution that you aren't aware of. Taking on a redesign means you will need to re-discover the things that were already solved. Along the way, you are bound to introduce new issues. There's a good chance you will create new production bugs. And you want your manager to sign up for that time, cost, and risk with zero concrete benefit? Do you understand why that's not a good choice on their part? Being non-technical actually makes it more likely the manager will allow you to do this, not less. It's much easier to convince a non-technical manager that an unnecessary rewrite is worth doing. A competent manager will know there's no virtually no upside to letting the junior developer rewrite the veteran's working solution.

Now, let's say the current implementation is buggy, slow, and/or unstable. Maybe the solution is brittle and making a few changes will take longer than a full rewrite. That's a different situation entirely. But you still might not win the argument: sometimes the cost of remediation is greater than the benefit. I've lost count of the times I've had to work within the confines of a bad design because a rewrite was not feasible or justifiable.

At this point in your career, you should be less worried about building it your way and more about learning from what others have done. It's not always going to be the case that you learn good ways to do things. My first few years as a developer were spent wrangling some pretty atrocious code within a poor design built with a dying platform. I hated it at the time but it was some of the best education I could have hoped for. I learned a lot about how not to do things and why good design matters. A wise person once told me: "smart people learn from their mistakes. Really smart people learn from the mistakes of others."

  • I'm not sure why you chose to write a sermon instead of answering the question. I know that if something ain't broke you don't have to fix it - I feel like I expressed at least some degree of self-awareness in the post (even just the fact I asked about it). I don't just want an excuse to just do things my way. My concerns were mostly about how to proceed with the code and I have since realised that the problem lays into in the use of the database but rather in the way it glues everything together in a very non-encapsulated way, making small behaviour changes require lots of changed code. – KubaFYI Jan 27 at 20:47
  • 2
    Q) "Am I justified in in feeling uncomfortable about this design choice?" A) I don't think so, no. Q) "I kinda feel like starting mostly from scratch only ripping out useful bits of the first MVP would be cleaner and take less time than working with the system created so far." A) Probably not. No. Q) "But is it just my younger-person-hot-headedness?" A) Probably, yes. – JimmyJames Jan 27 at 20:54
  • 1
    "My concerns were mostly about how to proceed with the code and I have since realised that the problem lays into in the use of the database but rather in the way it glues everything together in a very non-encapsulated way, making small behaviour changes require lots of changed code" That's a good question but it's not what you asked. Keep in mind that there are more options than 'full redesign' and 'keep everything the same' – JimmyJames Jan 27 at 20:59
  • that's fair enough, there there is a whole range of ways to proceed apart from those two extremes. I guess I'm just really want to avoid accidentally incurring a bunch of technical debt just because something is easy to do now. Either way, apologies if I sounded a bit snarky in my comment! – KubaFYI Jan 27 at 21:41
  • 1
    @Steve Sorry that was a rushed response. Learning from the failures of prior attempts is one of the main paths for human advancement. Today's unsinkable ship is tomorrow's Titanic. I see it all the time e.g. "I wanted this design to be super flexible so everything is represented as a string!" If you've been through the gauntlet of such a design, you don't need anyone to explain why that's a bad idea. TL;DR I'll conclude: the developers that I've worked with that have only written greenfield code and moved on are, in general, the worst: They have no idea that their shit stinks. – JimmyJames Jan 27 at 22:49
0

I can put myself in your shoes, and also in the contractor's.

Ask yourself the following questions:

First of all, how long will it take to change the design? Is it really worth it?

Effort in adding new functionality to the product.

Good points of the design as it is (this is the key as he has widely more experience than you, it could be designed for certain users, ...)

Have you had a handover of the product? Maybe that's what you need, you don't understand the product, motivation and its design.

It may not be so bad, just a little more info required

Not the answer you're looking for? Browse other questions tagged or ask your own question.