At a job that I recently started, I inherited some of the projects from the guy who previously held this position. One of the projects was a program that used a Website Platform's public API to get certain data from their site. The program itself is pretty simple, and is written correctly from what I can tell. It's not that complicated, so any big errors probably would have stuck out. When testing this program however, I get inconsistent results compared to their website. Ex. Searching using a keyword in the program yields 30 results and Searching using the exact same keyword on their official website yields 60 results.
After researching, I found that this was because the two searches use different internal mechanisms to return the results. Fair enough. I was thinking about approaching my boss about re-writing this application and using an approach where I query to website directly, get the HTML, find the certain class where the text I want resides and get it from there. It shouldn't be that hard (I know, famous last words...) and it would help our results be more accurate and more extensive.
Before I could approach him about this, he said that he wants to do the same kind of thing, but with another website with a similar platform. They have an API as well, but I'm wondering if we're just going to be running into the same problems here as well down the road.
My boss is not a programmer and doesn't know about the specifics of APIs and web queries, so that's why I'm asking here. Is it a bad idea to not use someone's public API, and instead just process the raw HTML for the data I want?