Tertiary to this question, I have been building my own imageboard that prevents [for example] duplicate images from being downloaded again and again on behalf of the client. How I do this, is that I keep all files in a database with a key being a hash of the file. The client sees the hash, and first checks its database to see if it has been downloaded before actually making a request. Similarly for my server; I also prevent duplicate uploads by having the client send me the hash first.
I am expanding a more general purpose networking library for downloading files from the web, and to my dismay; I discovered that not all servers will supply me with some sort of hash.
In an effort to de-duplicate downloads, and to continue partial downloads in which their url has changed, is there a way to reliably fingerprint a file from its headers and url?
Just taking an example here, of a plain HEAD request
QVariant reply->header( QNetworkRequest::ContentLengthHeader ) int 44374 QUrl url scheme() : https userName() : NULL password() : NULL host() : i.imgur.com port() : -1 path() : /oEdf6Rl.png fragment() : NULL query() : NULL https://i.imgur.com/oEdf6Rl.png QNetworkReply* reply Connection keep-alive Content-Length 44374 Last-Modified Sun, 21 Feb 2021 15:14:36 GMT ETag "83c16cca4ee371145485130383104315" Content-Type image/png cache-control public, max-age=31536000 Accept-Ranges bytes Date Fri, 26 Feb 2021 04:14:22 GMT Age 392375 X-Served-By cache-bwi5134-BWI, cache-yul12821-YUL X-Cache HIT, HIT X-Cache-Hits 1, 2 X-Timer S1614312862.217094,VS0,VE0 Strict-Transport-Security max-age=300 Access-Control-Allow-Methods GET, OPTIONS Access-Control-Allow-Origin * Server cat factory 1.0 X-Content-Type-Options nosniff NoError Unknown error
The only things that seem static here, are the Mime Type, and the file size. One thing I would be willing to do is do a
Accept-Ranges Download of certain bits, as I have found most servers do support this header, and from there; create a hash of the corresponding bytearray, and fingerprint it that way.
However, I am skeptical whether this would work reliably, especially concerned with something like two image frames that are nearly identical, but are in fact, not.
Am I pursuing a lost cause here? Or is there a reasonable way to fingerprint a file hosted on the web, without having to fully download it?
I'd like to do this with any file above 1mb large, because I have an exceptionally slow connection at times. Thanks.