Ratings Summit in Athens

Ratings Summit in Athens

A preliminary report by Jeff Sonas

On June 11-12, FIDE held a special meeting in Athens, Greece to discuss the implications of changes to the FIDE rating system, especially the increase of the K-factor.  The K-factor controls how rapidly a player's rating responds to their recent results.  The increase had been previously agreed to by the General Assembly, and was scheduled to go into effect at the start of July.  Ratings experts from around the world were brought together to recommend a course of action to the Presidential Board (which met a few days later in Krakow, Poland), and FIDE Deputy President Georgios Makropoulos chaired the two-day Athens meeting.

Meeting participants (left to right): FIDE Executive Director David Jarrett, FIDE Deputy President Georgios Makropoulos, Nick Faulks, GM Bartlomiej Macieja, Jeff Sonas, GM John Nunn, FIDE Qualification Commission Chairman Mikko Markkula, Vladimir Kukaev.  Not shown (because he took the picture): FIDE Treasurer Nigel Freeman.

I should say first of all that I was very impressed with the FIDE decision to hold this meeting, as well as their conduct during the meeting.  Whatever the procedural problems in the past that led to this situation, emergency discussion and analysis were certainly called for at this juncture, and it was a very productive meeting.  I have nothing but good things to say about FIDE throughout this process.

One point to emphasize is that for this meeting at least, the only two options were to recommend that the Presidential Board accept, or reject, the specific decisions of the General Assembly, so we did not spend too much time on debating the "perfect solution".  After analyzing the issue at length during our meeting, the majority clearly felt that the doubling of the K-factor was not necessarily an improvement, and that the matter was sufficiently unclear as to require further analysis.  In fact my personal feeling, after looking at the data and running many simulations, is to go beyond that statement and say that doubling the K-factor could quite likely be a very poor move.  Therefore we recommended to the Presidential Board that it refer the K-factor-increase back to committee, while supporting several other changes to the rating calculation, including the move to 6 lists per year (rather than 4 lists per year) as well as increasing the "350-point-rule" to be the "400-point-rule" instead.

I am currently working on a longer writeup summarizing the technical discussion, as well as additional analysis that was inspired by the brainstorming during the meeting itself and subsequent informal discussions afterwards with other attendees.  For now I would just like to point out a few highlights from the meeting:

1. It might seem strange to have supported the move to 6 lists per year, while at the same time rejecting the K-factor increase.  You would think these two decisions in combination would actually make the rating system less dynamic rather than more.  This is true, but it is important to recognize the relative magnitude of the two factors.  A change to the K-factor makes a big difference, whereas changing from 4 lists/year to 6 lists/year makes a very small difference.  Even increasing the K-factor for established players from 10 to 11 would be a severe overreaction to the move from 4 lists/year to 6 lists/year.  I looked at the average rating change for each player across various rating formulas, and my calculations indicated that the proper increase in K-factor, to account for the reduced rating "responsiveness" associated with the move from 4 lists/year to 6 lists/year, would be to increase the K-factor from 10 to 10.2.  So if we decide to increase the K-factor significantly, it ought to be for reasons other than the move from 4 lists/year to 6 lists/year.

2. I felt a particular responsibility for this situation because I advocated moving to a universal K-factor of 24 seven years ago, and this conclusion was used to partially justify the decision to double the K-factor.  My earlier analysis included optimizing a single universal K-factor, rather than the three-tiered K-factor that FIDE currently has (where some players have K=10, some players have K=15, and some players have K=25).  In fact the vast majority of players do have K=15 already.  Additionally my earlier analysis was performed on a smaller, unofficial subset of games from 1994-2001, whereas my more recent analysis covered the entire dataset – all players and all results – for the whole FIDE rating list from 1999-2009.  My older model also did not have a mechanism for introducing new provisional players into the system, whereas my latest model actually matches just about all of the FIDE rating regulations.  For all these reasons I feel that the recent analysis is more reliable, and I am very grateful that FIDE was so forthcoming with the data for my analysis.  I will be writing much more about my conclusions regarding the K-factor as well as my simulation model itself.

3. GM Bartlomiej Macieja held the consistent view that the ratings are so important, being used for direct qualification into the Candidates and World Cup events, that if we have to make the calculations somewhat more complex and it results in more accurate ratings, this is a good tradeoff.  For instance, two players may be so close in their ratings that their positions might be reversed if we adjusted for number of games played with White vs. Black, or if we used a different K-factor.  We discussed this at some length informally after the meeting, and others felt that the rating system is a nice balance right now between simplicity and accuracy, and were concerned about the introduction of further complexity into it.   Certainly another way to tackle this problem would be to change how the qualification process works, to make it depend more upon direct over-the-board results and less on the precise rating calculation.  This is a very difficult issue.

4. GM John Nunn had surveyed several of the top players, including both younger and older players from the top ten, and shared some of the results of that survey.  They all seemed to feel that the current system was working fine.  Obviously it is in their interest to be conservative because their ratings are already high, but some of these players (especially the younger ones) are still improving, and thus it would also seem to be in their interest to support a more dynamic system.  Nevertheless they pretty much all said that they didn't see a reason to change the K-factor.

5. GM Dmitriy Jakovenko had previously written an article for the ACP website regarding the increase of the K-factor, and this article was discussed extensively during our meeting.  His description of various K-factors as reflecting various beliefs in how important your last 20 games are, compared to all previous games, was quite useful in conveying the impact of the K-factor.  It was a very helpful perspective and provided a subjective assessment that people could make themselves, instead of just having to trust the analysis of people like me with all our statistics.  The subjective assessment shouldn't be the only one but it absolutely helps in framing the discussion.

6. Deputy President Makropoulos did much more than just run the meeting, and had several insightful points that motivated me to reassess some of my analysis. For instance he didn't completely agree with my approach of evaluating the "accuracy" of rating systems by comparing expected score against actual score, because we expect improving players to outperform their rating and thus perhaps some of that should be considered "improvement" in the players rather than just error in the rating itself.  He was also resistant to the idea that we must look back to Professor Elo's writings and opinions to determine the ideal course of action, preferring instead that we start from the assumption that the current situation is the default that players are accustomed to.  I agree that the burden of proof should fall upon people advocating a change in the current system, even if that change would involve moving more toward what Elo himself originally envisioned.

7. Finally, one topic that received considerable discussion was "rating inflation" – what does it actually mean, is there evidence of it, and what causes it?  This was relevant to the K-factor discussion because a higher K-factor does seem to increase "inflation", depending on your definition of the word, of course!  In addition if top players' ratings are really increasing too fast then it could devalue the Grandmaster title, which is based upon an absolute rating cutoff.  Inflation was certainly a topic where there was some disagreement among the experts regarding the fundamental answers to all three of the above questions, and certainly requires further investigation.  I will be writing much more about this, and I anticipate lively discussion among the mathematically-inclined readers in particular.

   
   Jeff Sonas in Athens

Press release

K Factor Meeting in Athens

Prior to the Presidential Board in Krakow, a meeting was held in Athens to discuss the proposed changes to the rating system and in particular the increase in the K factor. A small group of experts gathered to discuss the matter and to make recommendations, if needed, regarding the decisions taken in Dresden.

The two day meeting was chaired by the FIDE Deputy President, Georgios Makropoulos, and included the FIDE Treasurer Nigel Freeman, FIDE Executive Director David Jarrett, FIDE Qualifications Commission Chairman Mikko Markkula, FIDE Qualification Commission Councillor Nick Faulks, Jeff Sonas from California, GMs John Nunn and Bartlomiej Macieja plus Vladimir Kukaev from the Ratings Office in Elista.

The meeting focused on the K factor issue but also dealt with a number of other matters including possible inflation in the rating system. The meeting supported the move to 6 lists per year and the increase of the ‘350 point rule’ to ‘400 point rule’ but felt that the increase in the K factor should be referred back to the Qualification Commission and this recommendation was agreed at the Presidential Board.

Jeff Sonas and Bartlomiej Macieja both produced interesting and detailed material and John Nunn had carried out a survey of top players. FIDE is indebted to them for their hard work in the preparation for this meeting. In addition, the ACP forwarded an important paper by GM Dmitriy Jakovenko.

It is intended that this group continues to cooperate and meet again next year.

Source: FIDE

The great K-Factor debate on ChessBase.com