I have taken a look at Google Shopping for Hakko FX888D-23BY Soldering Station, and it gives this:
Notice that Walmart is $115.19 here.
Meanwhile, at Walmart.com it looks like this:
So it's $97.79 here. So from what it seems, Google is getting it wrong. They are likely using outdated data.
Also, Google Shopping doesn't have the Amazon equivalent, so for some reason it's missing that, maybe this is all automated (the linking of the scraped data) somehow, and they just missed it. I don't know.
But my question is if it is at all possible to keep a local cache like this Walmart price in Google Shopping up to date. That's the crux of the question. How possible it is.
In a simple thought, you can imagine that Google Shopping just listens for events whenever Walmart.com updates its price for a particular product. Then Google Shopping would just copy it and boom done. But this is the cloud and there is probably no direct connection between the two services. For the purposes of this question, I am assuming that Google is somehow simply scraping Walmart.com for the pricing data, rather than say getting some special access to an internal API or something. This way I can learn how to solve it in a practical way with practical constraints.
To phrase the question differently, I am wondering how fast something like Google Shopping could sync the data. Given that they probably have thousands of sites they are crawling, each with potentially millions or billions of pages, then they would need lots of computers. But I don't think it would be enough computers to be able to check the entire web for updates ever 5 minutes or so. If I remember correctly I read somewhere that it takes Google a few weeks to crawl the web, and they did it on some sort of rolling basis. Let alone trying to crawl the web and look for changes every 5 minutes, or 5 seconds!
So my question then is how do you go about getting a rough estimate of the amount of time it will take (given x computation resources available to use) to update every value in every corner of the app. This would lead to an answer saying roughly what the realistic timeframe is for syncing two systems at scale. And related to that, what a general architecture is for syncing the systems most efficiently (at a high level).