I have taken a look at Google Shopping for Hakko FX888D-23BY Soldering Station, and it gives this:

enter image description here

Notice that Walmart is $115.19 here.

Meanwhile, at Walmart.com it looks like this:

enter image description here

So it's $97.79 here. So from what it seems, Google is getting it wrong. They are likely using outdated data.

Also, Google Shopping doesn't have the Amazon equivalent, so for some reason it's missing that, maybe this is all automated (the linking of the scraped data) somehow, and they just missed it. I don't know.

But my question is if it is at all possible to keep a local cache like this Walmart price in Google Shopping up to date. That's the crux of the question. How possible it is.

In a simple thought, you can imagine that Google Shopping just listens for events whenever Walmart.com updates its price for a particular product. Then Google Shopping would just copy it and boom done. But this is the cloud and there is probably no direct connection between the two services. For the purposes of this question, I am assuming that Google is somehow simply scraping Walmart.com for the pricing data, rather than say getting some special access to an internal API or something. This way I can learn how to solve it in a practical way with practical constraints.

To phrase the question differently, I am wondering how fast something like Google Shopping could sync the data. Given that they probably have thousands of sites they are crawling, each with potentially millions or billions of pages, then they would need lots of computers. But I don't think it would be enough computers to be able to check the entire web for updates ever 5 minutes or so. If I remember correctly I read somewhere that it takes Google a few weeks to crawl the web, and they did it on some sort of rolling basis. Let alone trying to crawl the web and look for changes every 5 minutes, or 5 seconds!

So my question then is how do you go about getting a rough estimate of the amount of time it will take (given x computation resources available to use) to update every value in every corner of the app. This would lead to an answer saying roughly what the realistic timeframe is for syncing two systems at scale. And related to that, what a general architecture is for syncing the systems most efficiently (at a high level).

closed as too broad by Robert Harvey Jun 8 at 17:09

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • e-commerce sites upload their product catalogues to google. – Ewan May 9 at 11:33

There is a bit of a bait and switch here.

Google doesn't scrape the data to produce Google Shopping. The sites upload their catalogue : https://support.google.com/merchants/answer/7439058?hl=en

You are not seeing search results, you are seeing paid advertising


A few points...

You say "scraping". This word has a negative connotation. Hackers scrap website for email addresses so they can spam you with crap you do not want from their really cheap customers. What Google does is «spidering».

From what I understand, Google only uses a poll mechanism. In software you have two main types of discovering events: Polling (generally considered bad) and Pushing (usually viewed as really good.) I think Google turned off all the Pushing because it would give people who understood how that worked an advantage over others and it was probably detrimental to the search results (i.e. pages with little value.)

The search engines has many parameters. I will talk about the main one which is called "Page Rank". Google does not publicize that information anymore, but that doesn't mean they don't still calculate a rank to know what is more relevant. A page which gets updated often (every few minutes) on a pretty much constant basis and with good information (a.k.a. a newspaper front page, stackoverflow.com, etc.) is going to be polled very often by Google.

On the other hand, websites where some pages nearly never (if ever) change, will rarely be checked for changes.

So, with that information we can assume that Google probably has groups of computers for polling differently depending on a page's rank, how/when it changes, the type of expected data on the website (sports? movies? religious?), etc. We could imagine some 100 such groups...

One group would be for news, another would be for new websites, yet another for polling websites that nearly never change. Either way, they are going to do a poll, although you can still offer an XML sitemap file for your website, that will only help in finding all the public pages quickly, but in terms of how often the pages will be checked, the same algorithm as above will be used.

And yes, Google is pretty much the only one company (That I know of) that can poll websites world wide as quickly as they do and yes, in a way, their index is pretty much always out of date.

Statistics: according to info.cern.ch there are around 1.9 billion websites on the Internet. One for every 3 people... :-)

Not the answer you're looking for? Browse other questions tagged or ask your own question.