I have a system in which some data is inserted into database. Whenever that data is inserted subscribers should be notified about that (it doesn't have to happen just after inserting, but there is a time limit so I can wait like 10 s and then notify about X number of data inserted).
Problem is that each subscriber may have a filter. So lets say that new record was inserted to database and it is a product of type: shoes. I have 1 k of subscribers who are interested in that so they should receive email with that information but some of them have filter so that only if it is size 10.
What would be the best design for that to make it scallable?
Right now I have a process that keeps on looking for new records in database and for each one it iterates through all subscribers trying to find out if this user should be notified and if yes I send message to the queue with record and subscriber ID so that another component will receive it and send to that user.
This way I can have X number of components looking for new data and Y number of components receiving events from queue and sending notifications to users so it can be scalled out but still I think that it is not the optimal way for doing that.
Thanks for help