News Realms Remove Ads - Go Premium! Sign In or Register

Bluetracker

Tracks Blizzard employees across various accounts.

The MMR Hell debate continues: Nostromia shows Blizzard's algorithm is too slow to recognize improvement

A day ago, we saw the post about Nostromia's quest to take a potato Bronze 5 account to Grandmaster.

I've been following her progress, and I've noticed that even after 80 games with >90% win rate (twice as many games as the account started with), she's still getting a massive negative rank adjustment. What this means is that Blizzard's MMR still believes she has potato-level skill, and she doesn't deserve to be in Gold 4.

I didn't really believe in MMR Hell before, but that was on the assumption that Blizzard's MMR algorithm was working correctly. Looking at her hotslogs MMR, that algorithm now believes that she's at Platinum level. Hotslogs MMR might be a bit fast to update, but Blizzard's algorithm seems unreasonably sluggish in comparison.

This is a very extreme example, obviously. Imagine a typical player with 100s or 1000s of previous games that only manages a 55% or 60% win rate. How slowly would that person's MMR progress?

There is a possibility that there is some kind of "smurf cheat safeguard" that would recognize an extreme anomaly like Nostromia's account, and neglect to update MMR for some reason. However, I can't think of a reason why Blizzard would do something like that. Edit: Somebody mentioned cheating, and I agree that would be a legitimate reason to fail to update MMR.

I'm hoping that Blizzard is paying attention to this. If there is a real problem, I hope they will consider making changes to loosen up the algorithm a bit.

Edit: I want to make this clear because I feel like some people are missing my point. This post is not about Hotslogs MMR. Hotslogs MMR is unofficial; it is not the MMR that Blizzard uses. This post is about Blizzard's hidden MMR, which seems as though it is not updating to reflect the improvement in play on the account.

  • Loesby
    Posted 8 years, 2 months ago (Source)

    To put it simply, one player consistently beating players in Bronze League means the player should rise, but it does not mean the system should move them immediately to Platinum. There is also talk of the seeming mismatch between Rank changes and MMR changes, indicated by Nostromia's consistently negative Personal Rank Adjustment which, as we've said before, is intended to push players to keep their MMR and Rank closely in sync.

    These are the two things I'm most concerned about.

    Given what I have seen, I would argue that your model is not aggressive enough in placing players appropriately. There is too much conservatism in the model, and this leads to a slow, grinding experience to move up in rank and MMR. This then leads to a bad experience for all players involved, because players can't be matched appropriately. Bear in mind, this is just my perception, and I don't have your data to back it up, but I sincerely hope that you look into this across the spectrum of players to determine what the experience should be like.

    The concern is not that a player like Nostromia would fail to be placed in Platinum eventually. The concern is that she wouldn't be placed in Grandmaster eventually, because that is where her skill actually is. I would expect Blizzard's MMR to be more like Hotslogs in this respect, recognizing the ridiculously unlikely win percent, and moving the player up in ranks appropriately. Hotslogs has Nostromia at Platinum now, which is appropriate given the history of the account. Your model has her far below that, which suggests that it is moving too slowly. Once again, I can only speculate given the information that I have, but I hope you look into this experience across the spectrum of all players.

    Others have suggested that you look into the possibility of skipping ranks for high MMR accounts, and I hope you look into that as well.

    I appreciate the work put into this!

    I appreciate your interest! On a philosophical level, I agree with your speculation that Nostromia's internal rating should be increasing more rapidly. Improving our ability to place players correctly and quickly, and respond to improved skill, is one of the Matchmaking team's top priorities.

  • Loesby
    Posted 8 years, 2 months ago (Source)

    Hello, everyone!

    There's a lot of interesting discussion going on in this thread, and I've been glad to read a lot of it. I'd like to clarify a few points to help you all understand some of what Nostromia is seeing on this account.

    When Nostromia started playing seriously with this account, her MMR was in the bottom 0.01% of the Hero League population of Europe. As of this morning, when I started looking into this case, the account's MMR has risen to the 45th percentile. This is a massive improvement given the initial placement for her experiment. However, the Matchmaker has been performing admirably, according to the information it had at the time. Each of Nostromia's matches during her rise has been with and against players of similar MMR and Rank to her account's current rating, with very even win probability predictions. Because each of these matches looks even according to the available data, the Rating Update system will think "This player won in a game I thought was even. We'll increase her rating a small amount, since we were pretty sure she belonged here, and see how it goes with and against slightly better players." Winning and losing streaks are not factored into this update, because streaks of this length and consistency are extremely rare occurences, and honestly only happen when there is an extreme change in quality of play for a single account.

    To put it simply, one player consistently beating players in Bronze League means the player should rise, but it does not mean the system should move them immediately to Platinum.

    There is also talk of the seeming mismatch between Rank changes and MMR changes, indicated by Nostromia's consistently negative Personal Rank Adjustment which, as we've said before, is intended to push players to keep their MMR and Rank closely in sync.

    By design, visible Ranks change more quickly than internal Player Ratings. Going on a winning streak is meant to feel good, and will get a player promoted, many times to a Rank that is a little bit higher than the player's normal skill. The length of Nostromia's winning streak is very much an outlier, which exposes the different update speeds much more visibly than players who stick to one account and slowly improve their skills will see. Once Nostromia reaches a rank with other players of her true skill, she will start winning and losing games at a more even rate. At that point, the two different systems will converge on each other.

    I've watched Nostromia's stream for a while today, and it is very clear that her play is not that of someone in the lower half of our player base. We're actively researching ways to learn from players' individual performances, in order to move them to where a player of their skill belongs, more quickly. I can't promise timelines on this, or even that the research will bear fruit. We hope to be able to share news on this soon.

    Thanks to everyone in this thread for the discussion and interest! All of your feedback is enormously helpful. We will continue to monitor the health of these systems, and work tirelessly to improve your experience as players.

    Stay Tight!

    P.S. As a side note, I do want to mention that we don't condone players intentionally losing games to lower their rank, nor do we condone account sharing. Intentionally losing games creates a negative experience for 9 other human beings in the game. Account Sharing is a violation of the Battle.Net End User License Agreement. The best thing players can do for themselves, and for their fellow players, is to play on their own account, work hard to win every competitive game, and treat all other players with respect and kindness.

Tweet