How to modify ELO algorithm when number of games is low to reduce outliers

Question

I've developed the standard ELO implementation that can be found here, but the problem I'm running into is that my players will only have between 10 and 40 games in a "season". Since my first season has already completed (I'm just now starting my OWN rating system for my game), I know what the rank of my players ELO ratings should be, or at least very close to that. My problem is, after implementing that base algorithm, I have a some outliers that are way off.

Since I only have 10 to 40 games, my first thought to remedy this situation was to implement a loop around this process, taking the end result ELO rating from one cycle, and use that as the starting point for the next cycle (for each individual player). Then, repeat these cycles until the average or max deviation of ELO ratings from previous to current cycles is not greater than X. This helped to resolve some of the big outliers. There are still some smaller outliers that I'm still not comfortable with...

As an example, I have 7 divisions, for division 1, the guy that should be ranked #1 based on his ELO, ends up being ranked 5th, for division 2, the guy that should be ranked #1 is 3rd. The top 10 or so for each division are actually pretty close, except for these kinds of outliers.

How can I modify my algorithm, or what approach can I take to try and reduce the outliers?

Side note #1. For division 1, the guy that should be #1, that ends up #5, is actually ranked #1 for the entire season by the other rating system. I just don't know how that one is calculated (and it may not be calculated, it could be based on, I know x player is this good, so he's ranked #1).

Side note #2. I have made a modification to the algorithm. I want to "weight" the wins/losses by how dominant they were. If player A beats player B 20-0, I want that weighted more heavily, especially if they ratings are very close, but if their ratings are further apart (where a is already greater than B), then the ELO change shouldn't be that much anyway since they were expected to win in the first place. So, what I did was instead of using the k factor/value of 400 like the original formula uses, I changed it to be 100 + ((score difference) * 2). So the max k factor would be 140 (20-0 is the biggest win, multiply by 2, add to 100).

What other strategy could I implement to include weight results, as well as correct some of those outliers?

---

Without having the data in front of me (everything is at home), I'll try and sum up the scenario...

Division I

#1 Elo Rating of 2250
#2 Elo Rating of 2200
#3 Elo rating of 2190
#4 Elo rating of 2175
#5 Elo rating of 2170

Where #5 guy just beat the #1 to win the tournament, and his "body of work" seems more complete than the #1 guy (aka he played against, and beat stronger opponents than the #1 guy did). Maybe it's because the "weighted" k factor is throwing this off because #5 had lots of wins that were by only 2 to 5 points, where #1 wins were by a higher margin, say 5 to 10 points.

Maybe I just need to play around with my "weighted" k factor to see if I can get my calculated results closer to the actual results...

Answer 1

Here's what I've done to try and solve the scenario of the outliers. Re-simulating my entire season, knowing what each player SHOULD be ranked at the end of the regular season, and season ending tournament. Obviously when there are "upsets" in the season-ending tournament, it won't come out exact, but I feel this gave me the closest results that work for ALL players.

private double GetKFactor(double winnerRating, double scoreDifference, int winningPlayerMatchCount)
    {
        var kFactor = 32.0;

        if (winningPlayerMatchCount >= 11 && winningPlayerMatchCount < 21)
            kFactor = kFactor * .75;
        else if (winningPlayerMatchCount < 11)
            kFactor = kFactor * .5;

        return kFactor + scoreDifference;
    }