1

I noticed this pattern in my programs: I'm building a system, like the one I'm building for my local hackerspace right now. It's using database as its "brain" - a scraper as a separate service fetches building rental advertisements and a chat bot announces them on a task-related group chat. Both synchronize using the database.

The problem is, that the API we use to communicate with the chat requires me to keep track of one or more variables that need to persist across reboots and service restarts for the bot to operate properly. They don't really have much in common with each other, but it's tempting to just create a "bot_inner_state" (key TEXT PRIMARY KEY, value TEXT) table and be done with this.

At the same time, it kind of feels like bad design - especially when there's just one variable to keep. Are there any other sensible options? I assumed that since I have one mode of storage for non-configuration data, it would make sense to use it for everything, but basically every option feels like a compromise of sorts.

1
  • what kind of data would you be putting into this KV store? From a database standpoint that looks like a fairly bad fit, and I suspect this is more of a modelling problem? Perhaps some of these variables do form proper relational data when looking at it differently? Perhaps this isn't state, but configuration?
    – amon
    Jun 24, 2020 at 19:56

1 Answer 1

2

There is nothing inherently wrong with persisting chatbot related data in a relational database. It helps to think of a database as not a "brain" of the app and more of a means to persist important data that your application needs. If your chatbot app needs this data persisted then of course you could use a database for this.

I think though that your issue you are seeing is probably that you are coupling your applications together far too much. This is probably what your intutition is picking up on.

You should try to decouple your application components so that they don't share things like database schema objects. Ideally your scraper component and chat bot should not be using the same database objects. So then that would leave you with the problem that your chatbot would require the data that the scraper collected, so how should the chatbot fetch this data if not through the scrapers tables? This question highlights that your design is probably incomplete. There is an additional component that you would need to help enable the proper relational persistence of scraper data and to give a well defined API/interface that the chatbot can access.

I would define my components and responsibilities below to decouple everything:

  • Scraper - Collects data and stages it to database tables or files for processing.
  • Data Processor - Fetches data from stage tables and performs logic and transformations if necessary. Stores the data in relational form for use.
  • Data API - Provides CRUD operations against data in relational tables through an API or interface.
  • Chatbot - Utilizes the Data API interface to request data and perform chat functions. Owns its own database table(s) for persistence of chat bot specific data. This is a "thin" layer that contains no Data specific business logic and is focused entirely on being a dumb chatbot.

This provides a much cleaner design that is decoupled.

4
  • Thanks! I'm not convinced yet though. It sounds like you're replacing database with "data API" and putting logic in another service. I guess that my question is: what is the advantage here? Why is it bad to share database tables between services?
    – d33tah
    Jun 24, 2020 at 12:42
  • I would say that the disadvantages of tight coupling are that it can create maintainabilty issues and support issues. If you are diagnosing the root cause of an issue, the breadth of your search will be the entire interconnected application. It also makes it harder to independently test components since you have a harder time isolating a specific component. Decoupled applications are easier to unit test and integration test for this reason because you can easily mock behaviors of the dependencies. Entire books have been written on this concept so I could go on and on.
    – maple_shaft
    Jun 24, 2020 at 12:57
  • Understood. What surprised me was just calling shared tables "tight coupling". When one service writes to a table and the other reads from it, is it really already too tightly coupled?
    – d33tah
    Jun 24, 2020 at 14:02
  • In other words: if you replace the database with an API, you make it something both ends are coupled to anyway. Why would an HTTP API be a better thing to couple with, rather than the database?
    – d33tah
    Jun 24, 2020 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.