I am building a web application crawler that crawls for HTTP requests (GET, PUT, POST, ...). It is designed for one specific purpose; bug bounty hunting. It enables pentesters to insert exploit payloads at specific parts of the HTTP requests.
When using the crawler I sometimes run in to the problem of crawling a lot of similar requests (e.g.
/article/3, ...). This is a problem since, if I know
/article/1 is not vulnerable, there is a big chance that
/article/3 are also not vulnerable. This is because they probably run the same code on the back-end (they only get a different article out of the database). I therefore do not want to crawl them.
Lets say my crawler crawled the URLs below.
https://example.ltd/ https://example.ltd/news/some-news-alias https://example.ltd/news/another-news-alias https://example.ltd/contact https://example.ltd/news/some-other-news-alias https://example.ltd/news/and-yet-another-one
Then I could assume that all other URLs that match the pattern
/news/[alphabet&dash] do not have to be crawled because they probably run the same back-end code.
However, lets say my crawler crawled these URLs.
https://example.ltd/ https://example.ltd/users/sign-up https://example.ltd/users/sign-in https://example.ltd/contact https://example.ltd/users/forgot-password
Then I cannot assume that all other URLs that match the pattern
/users/[alphabet&dash] do not have to be crawled because they probably do not run the same back-end code.
How can I decide (with an as high as possible correctness rate) which requests are similar as requests I have crawled before?
The request and response data (headers, body, ...) of all the previously crawled requests (in the crawling runtime) are available for analysis to decide if the current request is similar to previously crawled requests.
The solution does not have to work right away but can start working after enough information has been gathered (maybe after about 200 requests of a certain (possible) route have been crawled).
I thought about first detecting possible routes based on URLs and afterwards checking if the HTML structure/tree of a certain route looks similar across all requests with that route. However, this seems to be kind of difficult since HTML structures may vary if you have e.g. a comment section below news articles.