which is the best way to correlate and group similar articles?

I mean something like Google News, which groups under a single topic different articles from different sources.

I'm not interested in something much sophisticated, it is just important to be efficient in terms of speed and resources needed. I can live with a correlation only on short text like the titles of the articles, not the article itself.

I read a paper which talks about tuples of two or three words (best with three) to extract and use to produce a score to achieve the correlation, but it's way too much academic for me

  • 3
    If I had such an algorithm then I would probably write my own software around it and form a startup. :)
    – maple_shaft
    Nov 22 '11 at 12:12

You can use Lucene Term Vectors.

Here is a blog post explaining it in detail. http://sujitpal.blogspot.com/2011/10/computing-document-similarity-using.html

Lucene is an indexing library: http://lucene.apache.org/java/docs/index.html

  • Sounds interesting, I'll take a deeper look into it. I wonder if I could do the same thing with Sphinx Search... the version 2 should have something like term vectors.
    – spider
    Nov 23 '11 at 13:46

If you are using Solr then there is a MoreLikeThisHandler

And in case using Sunspot with it, then its even simpler

post = Post.first Sunspot.more_like_this(post, Post, Page) do fields :title, :body with(:updated_at).greater_than(1.month.ago) facet(:category_ids) end


LDA is surely a way to get your job done - however it is complicated and involves the article.

"correlation only on short text like the titles of the articles", the information correlated in titles are too little to construct any model

  • 1
    Depends on the titles, I guess. Titles of academic papers are usually detailed enough to be useful that way, but other than that the titles will be much too idiosyncratic to be useful. Nov 22 '11 at 14:29
  • The analysis only on the titles is not a requisite. I just thought it was less resource intensive.
    – spider
    Nov 23 '11 at 13:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.