2

I have a database containing just simple URLs. It is as simple as it sounds for now and URL can link to a website or a document(i.e. anything parseable to text).

Now I have a simple code which inserts records to database. The problem is that website/document may be actually the same, just:

  • Hosted somewhere else
  • Not available so linked from Google Cache
  • Not available so linked from archive.org
  • Page can be republished from another source
  • etc...

I would like to get some kind of fingerprint of a website/document and think of a way to do so.

What I have thought of:

I can rely on title

because even if content is published somewhere else or cached on some server it will probably will have the same title. That's ok, as title is usually short and doesn't consume a lot of space. Downside - this works only on website. Maybe a filename is appropriate for documents but these can be renamed too.

I can rely on keyword count

But the problem is that I just enter URL and no keywords nor do I want to do so. It is a simple URL storage engine.

I can get a some kind of checksum of all the content

But this method would be a total guess.

SO MY QUESTION IS: How can I fingerprint a content so later on I could identify possible duplicates ?

EDIT

I don't want to fingerprint just the title. I want to fingerprint the whole content. Content stays the same but can be hosted anywhere and it's structure(title too) can change. For documents - filename can change too. I want to fingerprint content. All the text so I can later on identify possible duplicates.

5
  • 1
    So you're looking to fingerprint just on the title/url? or can you fetch the document to fingerprint it (even if you don't store it)?
    – user40980
    Oct 8, 2013 at 14:09
  • Do the linked documents/files have anything in common that you are aware of or could they be anything?
    – glenatron
    Oct 8, 2013 at 14:10
  • 1
    Even if content is republished widely, there's going to be differences in the document as each site puts its own branding on it. Similarly, the title is likely to have the name of the site appended to it. I think you need to look deeper into the content and actual evaluate it for similarities to others via text comparison; but that's not cheap.
    – TZHX
    Oct 8, 2013 at 14:18
  • How do you define duplicate? Byte for byte? In that case, what's the problem with a hash? Oct 8, 2013 at 15:39
  • How big of a fingerprint are you willing to store? How exact do you want the fingerprint to be?
    – user40980
    Oct 8, 2013 at 20:15

1 Answer 1

3

The answer depends on how accurate you wish to be.

Title

Title would work with 99% of web sites, and you'd have the added advantage that it lies in the head, meaning if the server supports it, you could even only request the page header. The disadvantage would be that, of course, you may have the same title but different pages.

Keyword

Check by keyword would have the disadvantage that two similar pages may share the same keyword count (take into consideration the possibility of finding keywords in the menu and none in the main body -- all pages with that menu are "the same" for what concerns keyword count). Also consider that you'd have to download the entire page to do this sort of check, so it has no edge over the title method.

Checksum

Checksum would be the most accurate by far. Perhaps even too accurate than what you're expecting. All it takes is a server-generated timestamp and you've got a page different than the previous. However, if this is acceptable, then you can't beat a md5 checksum. It would also be relatively easy extract the checksum and to save in the database.

It has the added advantage that you can perform checksums on not just the html source but the images and stylesheets as well. You could perform a "deep" check on all page resources once a week and a shallow check once a day on the html source (you'll need separate columns in your database for each check type).

Conclusion

Of all these solutions, I would probably implement the Checksum. My database would house two new columns, one for a MD5 checksum and the other for date of page expiration. The checksum tells me how to know if the page has changed while the page expiration would tell me with what frequency I should check it, assuming it is available.

I hope that helps, and good luck!

5
  • Thanks! Thought about checksum(md5) and you explained it well but was very unaware about header part which looks promising for some cases. Oct 8, 2013 at 15:05
  • 1
    @lukas.pukenis If performance allows, go with a strong hash, such as SHA-2. Oct 8, 2013 at 15:40
  • 1
    @lukas.pukenis I agree with CodesInChaos. I mentioned MD5, but probably SHA-2 would be better suited. What's important is that you don't try to do the checksum yourself. ;)
    – Neil
    Oct 8, 2013 at 15:44
  • @Neil I have grown out of "i will make my own framework" age ;) Oct 8, 2013 at 16:22
  • @CodesInChaos thanks for suggestion about SHA-2! Oct 8, 2013 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.