1

I am making a watchlist type application that will get all items being watched and then poll an api every ten minutes or so to check on its status.

I am trying to design the database portion of this project and wanted some advice on the best technology to use. I am very experienced in relational databases but have used MongoDB for this project.

What I am thinking of doing is having two tables one watch_list table and one item_status table. One application can get a distinct list of items from the watchlist table and then poll the api and then update the item_status table when it likes, regardless of what's going on with the watchlist, can also multithread with the API. If the API returns a true status some action needs to happen. The other app simply takes requests and puts items in the Watch_list table. THe problem I foresee with this is that we are storing info we don't really care about in the item_status table, i.e. once the status is true then some action happens and we don't care about it anymore job done, no one can put that item back in a watchlist until it is false again so you could even just delete those records.

This leads me to the idea that perhaps I only need one table, which includes a flag to if the status has been returned as true and the action has happened, as well as the time it was last checked.Only reason I don't like this however is that I will be hammering this table with updates from calling the API to check if the item status is true so I'm worried about writing new items into this table also I like the idea that I can have a separate entity for polling and that can just run in the background updating the item_status table.

As well as this I'm trying to understand which technology would be best for this data, postgres or mongodb are my preferred two choices, my data is incredibly relational so I am finding it difficult to think of how mongodb would provide a benefit. As I am going to be updating fields a huge amount I would like to know which technology would be more suited to this kind of project.

Improve this question
1
  • From your second paragraph it sounds like you are already using MongoDB for this project. What version of MongoDB are you using? Instead of polling for updates you could consider using the Change Streams API (MongoDB 3.6+) to process items based on status changes.
    – Stennie
    Oct 17, 2018 at 23:29

1 Answer 1

Reset to default
1

As with all 'do something when x happens' questions the key is to capture the incoming event, rather than to watch for its effects.

Hide your database behind an API, in the Update methods of the api, check your conditions and trigger your event as required.

The database tech behind the api doesn't matter.

Improve this answer
2
  • It does matter though, if I want to scale this application, it really does matter, the DB is already behind an API. I'm going to be hitting this database very frequently thanks to the polling so I am interested in which technology would give me the better performance for this.
    – Simon Nicholls
    Oct 12, 2018 at 11:11
  • Based on the above would you recommend simply the one table approach?
    – Simon Nicholls
    Oct 12, 2018 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.