I have a scraper service that looks for media online. For every source type,
there are two tables. A common
source table and a
source_type_media table that are related by
source_type_media.source_id, this service will store raw information.
The scraper service will scrape at some frequency via a cron job, which will look up the
sources table, and then scrape media based on information from that source. This will get new data, as well as update old data, say updating the number of upvotes any one media has.
Note that all the data are in different media tables because each source's media may have different data structures and are stored in raw format, call it
The other service I have is my core app service, which the UI will talk to. This core service will make a request to the scraper service, which will normalize and aggregate the media data from the different sources. This normalized data is then passed back to the core app, which will be stored in the core app's DB.
At this point in time we lose the relationship between the source table as it's only related to the raw data.
I want to maintain this relationship because the flow of the UI is for users to first add sources they're interested in, and only the related media will show up on their feed.
I believe it would be more correct if we leave the normalized data in the scraper service, and the core app can proxy the request whenever needed, to the scraper service, however I have a caveat. The user should be to create comments on the normalized media, so that means the normalized data should be in a location these comments are located.
So the total flow is:
- User adds normalized sources (from
- User goes to media feed and sees related media to those sources (also
core app, all normalized)
- User can comment, like, or rate the media (these events are related to the normalized media table in the
- On a certain frequency, the the scraper service will scrape updated or new data
- On a certain frequency, the core app will get normalized data from the scraper service, both old and new will show up
- If user goes back to their media feed, the user should be able to see updated information on that media
Depending on where I put the normalized media, I have a different problem:
- Put it in the core app, and I've lost the relationship between sources and media
- Put it in the scraper service, and I won't be able to relate user comments/likes/etc to the normalized media
I'm thinking maybe I should put a UUID on each source and each media that will be the same across all my services, but this doesn't feel like micro-service architecture anymore.