Deploy a bot on your server to retrieve the pages over TLS, preferably using certificate pinning, and don’t worry about performance/load issues and optimization until much later.
Explanation:
For the use case you described, you’re going to need a trusted external entity, no matter what. This is because nothing that happens inside the user agent can ever be trusted, including the entire page content.
At the moment, your trusted entity is the the shop’s server code, or more precisely, the secret key buried inside it. The shop’s server code takes all the data you’re interested in and wraps it to a signed package, which eventually finds its way to your server.
Whatever it is that you choose to remove from that package, you will need to find some other trusted entity to prove it’s valid.
Your goal seems to be to simplify the API you’re offering to the shop server operators. You want them to never again have to feed e. g. a price change into your SDK, or a new product. Ideally, you also don’t want them to have to touch their client code that talks to your API; this is relevant when they exchange a moving part behind the scenes, e. g. their data store, or their shop software.
In other words, your goal is to remove as much product data as possible from your API/SDK the shop operators need to code against.
However, to fill this data gap, you’re going to need:
a way to look at the content of the shop’s generated HTML page,
and some trusted entity (out of the user agent’s reach) to verify that the page content is authentic.
Simply doing the crawling server-side over TLS has the following advantages:
Albeit being far from perfect, HTTPS/TLS gives you a working, time-tested, reasonably secure infrastructure essentially for free. As shops tend to already have TLS in place, there’ll be zero setup time on your side (or near-zero if you consider certificate pinning, which is advisable).
Data integrity will be less likely to be compromised, e. g. by mistakes in your SDK’s implementation or the shop’s client code using your SDK.
There will be a little less risk of key theft regarding the shop-side HMAC secrets; it is a sad fact that not all shop admins do their homework, but even then, they will still be far more likely to make critical blunders with a custom-built HMAC infrastructure, even with a bug-free one, than mess up TLS certificate handling, where at least knowledge and skill are more widespread and thus more accessible.
The previous point goes double when you consider how shop admins will always come and go, and there will be issues with lack of proper handover procedures or training regarding your SDK.
You won’t need to worry about distributing or updating secret keys to the shops in the first place.
Your data harvesting will work at least as robustly as it does today: you might be able to still crawl the latest data even when (not if!) a single link fails in the long chain of: shop’s SDK client code → your SDK → shop’s web server → user-side ad blockers and other browser extensions → JavaScript → user agent DNS → your server.
You can still leverage the existing JS scripts for user agents to trigger your server; you’re still going to need debouncing but at least you won’t have to worry about cron-scheduled crawling.
The drawbacks would be:
An increase in traffic and associated cost between the shop server and your server. I’m not experienced enough in operating a shop to estimate what this would mean; however, I do know that there are numerous ways to mitigate such costs – that is, if those costs are even noticeable. You will always be able to tune your spider so it cuts down on fetching irrelevant resources, e. g. non-inline JS, CSS, navigation, advertisements, or static images. The good thing is, you don’t even have to do either of this unless someone complains or your bill increases, which might not even happen for a long time, if ever at all. It does not even stop there; you could look into alternative server hosting to get cheaper downstream bandwidth. You’re free to place one or more forward proxies in front of your server. On top of that, you can leverage existing shop-server-side caching, timestamps or ETags. And of course, you can simply tune your spider so it debounces outgoing requests, or even falls back to scheduled requests if you choose so.
The initial spending for shops to change their infrastructure. This can be mitigated when you choose a soft roll-out, giving the new, thinner SDK only to new shops while leaving the old infrastructure in place for the existing ones until the time of their next scheduled redesign.
You need to develop and release an updated, thinned SDK for the shops that does exactly what you want (whatever it is that remains for the SDK to do; maybe think about doing away with it completely?).
The initial spending, in time and money, for you to develop and maintain your bot.