Using browsers to monitor changes in e-commerce pages

Question

I've been looking at ways to monitor for changes in things like price and availability on e-commerce sites via visiting browsers, with three constraints:

• the sites often don't have much ongoing development effort behind them, so the solution needs to be simple to implement
• there should be minimal impact on how quickly the page is generated and delivered
• it should not be possible for a third-party to spoof reports of changes

The approach I'm using at the moment is to give the site developer an SDK that allows them to generate an JSON object using the same data they're using to generate the page. They render this object into a script element in the page, along with an HMAC of the JSON using a secret key. The page also references a small script that pulls out that JSON and POSTs it to my server, which rejects anything that doesn't have a valid HMAC or has a timestamp too far in the past. This prevents most spoofing, except for a small replay window.

To make it even simpler for the site developer, I have considered using a microformat approach where they add attributes to declare things like "this element contains the price" and "this element says whether the item is in stock or not". Again, my JavaScript would be referenced in the page, but this time it would look for those attributes and build the JSON itself rather than pulling it out of one element. This means no extra work while generating the page, but it has the drawback that there's no HMAC, so it's easily spoofed. Am I right, or am I missing something?

One possible solution is to verify differences by retrieving the page directly, and parsing it on the server side to find those attributes, but I'd prefer not to increase the load on the e-commerce site unnecessarily.

Is there a solution that can use the microformat, but still have a reliable anti-spoofing measure?

Answer 1

Deploy a bot on your server to retrieve the pages over TLS, preferably using certificate pinning, and don’t worry about performance/load issues and optimization until much later.

Explanation:

For the use case you described, you’re going to need a trusted external entity, no matter what. This is because nothing that happens inside the user agent can ever be trusted, including the entire page content.

At the moment, your trusted entity is the the shop’s server code, or more precisely, the secret key buried inside it. The shop’s server code takes all the data you’re interested in and wraps it to a signed package, which eventually finds its way to your server.

Whatever it is that you choose to remove from that package, you will need to find some other trusted entity to prove it’s valid.

Your goal seems to be to simplify the API you’re offering to the shop server operators. You want them to never again have to feed e. g. a price change into your SDK, or a new product. Ideally, you also don’t want them to have to touch their client code that talks to your API; this is relevant when they exchange a moving part behind the scenes, e. g. their data store, or their shop software.

In other words, your goal is to remove as much product data as possible from your API/SDK the shop operators need to code against.

However, to fill this data gap, you’re going to need:

• a way to look at the content of the shop’s generated HTML page,

• and some trusted entity (out of the user agent’s reach) to verify that the page content is authentic.

Simply doing the crawling server-side over TLS has the following advantages:

• Albeit being far from perfect, HTTPS/TLS gives you a working, time-tested, reasonably secure infrastructure essentially for free. As shops tend to already have TLS in place, there’ll be zero setup time on your side (or near-zero if you consider certificate pinning, which is advisable).

• Data integrity will be less likely to be compromised, e. g. by mistakes in your SDK’s implementation or the shop’s client code using your SDK.

• There will be a little less risk of key theft regarding the shop-side HMAC secrets; it is a sad fact that not all shop admins do their homework, but even then, they will still be far more likely to make critical blunders with a custom-built HMAC infrastructure, even with a bug-free one, than mess up TLS certificate handling, where at least knowledge and skill are more widespread and thus more accessible.

• The previous point goes double when you consider how shop admins will always come and go, and there will be issues with lack of proper handover procedures or training regarding your SDK.

• You won’t need to worry about distributing or updating secret keys to the shops in the first place.

• Your data harvesting will work at least as robustly as it does today: you might be able to still crawl the latest data even when (not if!) a single link fails in the long chain of: shop’s SDK client code → your SDK → shop’s web server → user-side ad blockers and other browser extensions → JavaScript → user agent DNS → your server.

• You can still leverage the existing JS scripts for user agents to trigger your server; you’re still going to need debouncing but at least you won’t have to worry about cron-scheduled crawling.

The drawbacks would be:

• An increase in traffic and associated cost between the shop server and your server. I’m not experienced enough in operating a shop to estimate what this would mean; however, I do know that there are numerous ways to mitigate such costs – that is, if those costs are even noticeable. You will always be able to tune your spider so it cuts down on fetching irrelevant resources, e. g. non-inline JS, CSS, navigation, advertisements, or static images. The good thing is, you don’t even have to do either of this unless someone complains or your bill increases, which might not even happen for a long time, if ever at all. It does not even stop there; you could look into alternative server hosting to get cheaper downstream bandwidth. You’re free to place one or more forward proxies in front of your server. On top of that, you can leverage existing shop-server-side caching, timestamps or ETags. And of course, you can simply tune your spider so it debounces outgoing requests, or even falls back to scheduled requests if you choose so.

• The initial spending for shops to change their infrastructure. This can be mitigated when you choose a soft roll-out, giving the new, thinner SDK only to new shops while leaving the old infrastructure in place for the existing ones until the time of their next scheduled redesign.

• You need to develop and release an updated, thinned SDK for the shops that does exactly what you want (whatever it is that remains for the SDK to do; maybe think about doing away with it completely?).

• The initial spending, in time and money, for you to develop and maintain your bot.