I'm trying to create a nodeJS application. It allows users to rate a bunch of songs and it stores them in their user profiles. I use this information to compare them to other users, and try to find users with similar interests in songs, and I suggest them new songs based on that.

Each user profile basically looks like this

userID: [the userid of the user]
songs: [the list of songs that the user has rated]
ratings: [the corresponding ratings each user gave to the song]

Each song is represented by a 9 digit whole number and each rating is a whole number from 1 to 6. Basically put, I have to compare one user to the rest of the users to determine which of them match this user the best. By match, I mean gave the same songs similar ratings. To do this, I created a simple algorithm.

Step 1) Create a list, in which each entry maps our target user to each of the other 
Step 2) Now consider each entry, determine which songs both of the users have rated, and 
        store those songs (along with the corresponding ratings each user gave them) in 
        this entry itself 
Step 3) Now iterate through each entry and perform the following operations 

        a) let percentage = 0
        b) let num = [the number of songs that both users rated]
        c) iterate through each song (that both the users rated) and perform the
           following operations

                i)    determine the score our target user gave this song and store it in 
                      variable a
                ii)   determing the score the other user gave this song and store it in 
                      variable b
                iii)  map a and b to new values based on this 
                        1 --> -3
                        2 --> -2
                        3 --> -1
                        4 -->  1
                        5 -->  2
                        6 -->  3
                iv)   now calculate sum as 
                    sum = |a| + |b|
                    where || is the absolute values
                v)    now calculate degree as 
                    degree = sum/2
                vi)   now if a * b is less than 0 then
                            calculate (percentage - degree) and store that value
                            again in percentage
                          if not then 
                            calculate (percentage - degree) and store that value 
                            again in percentage

        d) now  calculate (50 + (percentage / (6*num))*100) and store that value 
           in this entry as match

Step 4) Now that I have my list of entries (along with the match between each pair 
        of target user and other user) I sort the list in descending order of match 
        and from that, I can determine which users have the closest match in taste by 
        selecting from the first entries

Now for each user pair, I am completely neglecting the songs that one user rated and the other didn't, and that is okay for me.

There are several problems with this method though, the biggest one being that this algorithm is very time consuming for a large set of users (say around 1,000,000). And also that I have to load in all the users, every time I need to find a set of matching users for just 1 user. And I need to do this repeatedly for that user, to update his/her list.

Is there a way to make this more efficient? Can I assign a value to each user that takes into account all the songs that they rated, and use that number to compare the users? Is that even possible? I guess what I'm asking is, how can I compare and contrast this data, mathematically, to find similar users, efficiently. Also, can someone suggest a proper tag for this type of question?


wigy posted a link to some stuff that is highly relevant but I think there are a few simple things that can help here. I'll offer some suggestions that are probably not optimal but might help you move forward.

The first thing I would suggest is to avoid iterating over every other user for one user. I would instead iterate through the songs that user has rated and then load all the users that have also rated that song (this might be problematic for very popular songs but well get back to that.) Count how many times each user comes up. For example, if a user has rated 6 songs that the target user has rated, they get a 6.

I would then drop any users that fall beneath some threshold (you'll probably need to try different things). Start with 10 maybe. The reason is that statistically, it means almost nothing if two users have both rated one song. The more songs they've both rated, the more confidence you should have in your calculation. This should also help with the problem of 'hit' songs. There are a few songs that lots of people have heard not that many that everyone has heard.

Another thing you can do is that once you have evaluated two users, save the answer. You haven't really explained how you are storing your data and a lot of how you solve this problem will be about persistent (or long-term) storage of your data and results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.