I'm utilising OpenCalais API to tag the articles from multiple news sources.

I know which category each article belongs to (e.g. crime, politics etc). Also each article has three social tags available.

How can I find out which topic is the most talked about by multiple news sources?

I thought I first gather all the saved articles in the past 24 hours.

  • Grab the first article and add its category (e.g. crime) as a key to a dictionary. The value for that key will be a list of articles.

  • Hence within a loop, I would add any article from that category to the list above.

  • With this approach I have a dictionary where the keys are the category and value represents the articles belonging to that category.


   "Crime" : ["article1", "article4", "article6", "article7",
   "Politics" : ["article2", "article3"] 

The challenge is to find out if the articles in Crime category are talking about the same crime or not.

e.g. article1 has these three social tags:  
   ["Crime in London", "Holborn", "Subterranean London"]
   ["Hatton Garden", "Holborn", "Subterranean London"]
   ["Clerkenwell crime syndicate", "Crime in London", "Holborn"]
But article7 seems to be about a different kind of crime than Hatton Garden heist:
   ["Subterranean London", "Tube", "Assault"]

I suppose I need to use some kind of mathematical intersection to find out for each article how many social tags match each other.

So that I could say article1 and article4 have two tags that match each other, and hence have a higher probability that they are covering the same news.

Article6 is similar, as it matches two tags with article1, but not article4. However because article1 and 4 match, we conclude that article1, 4 and 6 are covering the same news. (I don't know how to achieve this in code)

While article7 is matching only one Social tag, which matches article1 and article4 respectively but hence is less probable to be talking about the same kind of crime. (Unsure how to achieve this)

Does it make sense, what I'm trying to achieve? Thanks for advice.

  • 1
    This looks like a research project, and tackling the issues you are describing on the programming side doesn't sound right to me. You are basically trying to build a (fuzzy) classification system. (If this is a proof of concept, then just go ahead with whatever metric you can think of). Otherwise, you are jumping in the implementation of a method you have not designed yet. A proper literature review should point you in the correct direction (and then you'll have the correct keywords, not only "Python"). So, yes it makes sense, but no, not this way :) – Tibo Jan 19 '16 at 13:39

I suspect that judging articles to be about the same event based on three tags may not give you very good results.

If you have access to the full text of the articles, then you are better off comparing them by looking at the actual content. There are many ways of doing this. One would be to build a classifier, like Tibo mentioned in his comment. I suggest you start by computing tf-idf vectors for the documents and calculating their cosine similarity.

There is a similar question and detailed reply (with python examples on how to do this, and a link to yet another question on this) on StackOverflow: https://stackoverflow.com/questions/8897593/similarity-between-two-text-documents

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.