1

I have lists of data I'm trying to organize where the list with the least variation/most precision/closest to the average and most amount of data is most highly rated. Some of the lists are like 2000 numbers long and others only have 1 number. Obviously the arithmetic mean of a list with one number will be exactly the value of the one number. Any individual value from a list of 2000 numbers might not be as close to the arithmetic mean. When sorting these, all the smaller lists will be higher rated.

It seems really easy, but I can't come up with a way to order the list more proportionally.

Examples of lists:

List 1

12, 20, 15, 17, 9, 19

Average of list:

12+20+15+17+9+19/6=15

Score: (15-12) + (20-15) + (15-15) + (17-15) + (15-9) + (19-15)= 20

List 2

30

Average of list: 30/1=30

Score: (30-30) = 0

List 3

10, 10, 10, 10, 10, 10, 10, 10, 10, 9

Score: (10-10)...etc = 1

Ranking

  • List 2: Score=0

  • List 3: Score=1

  • List 1: Score=20

This is what i have right now. The problem with this is that list 2 will be at the top because its small. I'd like list 3 to be at the top because the score is small and it has a lot of members. I'm not sure how to add weight to size though. List 1 is an example of a list with a lot of members, none of which are really close to the average. Thats two bad things so it should be at the bottom.

As for outliers, they arent treated any differently. If one super high number gives a list a really bad score thats okay.

5
  • Could you give examples of the problem you are working on? Some example lists and how the lists are scored? Why are smaller lists rated higher? By average you mean mean (the mean? or something else?)? What approaches have you tried? How do outliers and suspected outliers factor in?
    – user40980
    Feb 8, 2015 at 2:23
  • 1
    Sorry about that. I updated the question Feb 8, 2015 at 3:41
  • Maybe you could sort on the count of number that are within x% of the average. Feb 8, 2015 at 7:11
  • 2
    To be frank I don't have a good understanding of what is the correct way to measure variance either, but maybe you can read about variance in general, on Wikipedia ...
    – rwong
    Feb 8, 2015 at 7:17
  • 1
    Second that. You want to sort lists by their combined variance and length, so compute those two and combine the scores however you want. Feb 9, 2015 at 12:08

1 Answer 1

2

A common approach to measuring what you're describing is to use the standard deviation of the set of numbers. Others prefer to use the variance instead. Fortunately, calculating both is fairly simple once you understand the algorithm.

For the variance and standard deviation, follow these steps:

  1. Find the average of your set of numbers:
float avgValue;  
float totalValue = 0;  
for( var i in setOfNumbers ){  
    totalValue += i;  
}  
avgValue = totalValue / setOfNumbers.Count();  // or use Length() instead ...
  1. Sum the squares of the difference from the average value.
  2. Find the variance by find the average from step 2.
//Find sum of squares of the difference from average
float sumOfSquares = 0;  
for( var i in setOfNumbers ){  
    float diffFromAvg = i - avgValue;  
    sumOfSquares += diffFromAvg^2; // use your language's square function  
}

//find variance
float variance = 0;  
variance = sumOfSquares / setOfNumbers.Count();  // or use Length() instead ...
  1. If you want the standard deviation, that's the square root of the variance.

This link and this link also offer up some additional explanation for calculating either the variance or the standard deviation.
You said that the longest list you have is 2000 numbers, so you're not likely to need alternative algorithms. That said there are a number of ways to calculate variance and standard deviation in one pass.
But as with all things in life, there are disputes over the best approach, so do your additional research if you get beyond the basic algorithms.

1
  • I've been working on this and I'll come back after tommorow after I've investigated standard deviation. I'm also considering matthew briggs response Feb 9, 2015 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.