0

(Hi I’m new here - I believe this question suits here from the guide saying “ architecture, and design” - however if there is a better place, just pop me a comment. As there’s no actual code it doesn’t suit stack overflow )

I am designing a simple site and app where people can create alerts for product stock availability and prices.

So users would create an alert for let’s say a lawnmower at $125.

There would be many products at many prices from a few different vendor APIs.

The hard bit is the backend. I’d have to loop through a unique list of all products submitted, query APIs for their price, put that in another table of product-to-price, then loop through the entire user table again checking that current price against the alert price the user has set.

But the alert needs to be real-time as users bid on products, yet that script could take minutes to run with just a few users. I’m hoping for many users with many notifications.

Any ideas on a better architecture? Open to tech, ideally JS and AWS but am open to change.

7
  • 2
    Are the Vendor APIs you have to poll or are they pushing to you? It seems to be me that your biggest challenge to be "real time" would be that you have to constantly poll vendor APIs.
    – Helena
    Jul 31 at 10:54
  • 1
    How many Alerts and how many Product offers to you expect to have per day?
    – Helena
    Jul 31 at 11:16
  • If your business is polling and checking data changes from 3rd party services "by brute force", then I agree with @Helena. It's hardly possible to make it real-time. Not mentioning if any of the APIs has hard SLA (e.g limited req per client over time, or downtimes for whatever reason). You will have to elaborate a bit more what are the key features of the application to determine if there's an alternative to your current "strategy"
    – Laiv
    Jul 31 at 11:27
  • 1
    @Helena yes it's polling vendor APIs - you're right in that there will be polls every minute to each vendor. That's many calls. the key feature is the realtime nature, it's got to be as quick as possible. I've been thinking more about a Kafka producer to poll the APIs and send events which a consumer could use to check the Database and send alerts but that first bit is still the hard part - all the polling
    – Wayneio
    Jul 31 at 13:28
  • Doesn't matter how you do the poll. Polling won't be ever real o near real-time. You need changes to be pushed from the source as close to the time they happened as possible. You could tho limit the polling to a very small set of products. For example, only those that were set for notification, rather than checking and inspecting everything whether you need it or not every x minutes. Minutes in computing science are perceived as unsatisfying ages by the end-user.
    – Laiv
    Jul 31 at 14:23
1

Based on the information, here is the system design I would start with:

Asynchronous background job

  1. At pre-determined intervals, poll the vendor API's - being respectful of rate limits.
  2. Add/update those products with the vendor data, e.g. VendorID, Price, etc., into a database such as Postgres.
  3. Add a dedicated secondary read-only follower DB that replicates data from the primary configured with PG bouncer to pool connections. This is because you mentioned that you would like the responses as real-time as possible - one way to "speed up" requests would be to ensure that the service has dedicated access to that database, and other services can't take up resources. This model is a general approach when connecting third-party analytics systems or tools that need to be read from your database.

Synchronous API

  1. When a request comes in that can tolerate the possibility of stale data between polling intervals (e.g. alerts) of prices serve the data from the secondary DB
  2. When a request comes in that requires the most up-to-date information, issue a request to the backend to refresh that specific product from the vendor API so you can return the most up to date version of the data. In this scenario, you could then display a loading interface back to the user while you make the request.

Considerations

When building a system that polls and stores data from an external source, you need to consider the following very closely:

  1. You can't guarantee the availability of vendor API's - how does your application handle this?
  2. How tolerant would your systems be to the stale data - from the second the response is returned from the vendor API, that data should be considered stale, and you should treat it as such internally.
  3. How can you optimise your schema design and SQL queries - using things like Aggregate Functions to perform as much computation on the database as possible (they are incredibly good at that).
  4. You might not need a full-blown DBMS such as Postgress, and maybe something more straightforward like SQLite might be more appropriate as it's incredibly fast for both reads & writes.

I'm happy to debate and discuss this further - let me know your thoughts (the good and bad)!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.