Are chess computers improving faster than grandmasters?

Jeff Sonas on Man-Machine, Part II:

Are chess computers improving faster than grandmasters?

In Part I we looked at the historical results of chess games played between the strongest grandmasters and the strongest chess computers. The surprising conclusion was this: top grandmasters and top chess computers are dead-even, and they have been stuck that way for some time. Neither side has actually won a match from the other in five years, and the last seven events between grandmasters rated 2700+ and chess computers have all been drawn. How long can this deadlock last?

You're probably thinking, "Okay, sure, they're even now, but isn't it inevitable that computers will just get better and better and eventually leave the humans behind?" Well, yes, computers will definitely get better, but don't forget that human players are improving too! Human players are constantly getting better at chess; it's just super-hard to measure this statistically, because improving humans play against other improving humans. It is NOT inevitable that computers will surpass humans.

Let's look at some numbers and graphs. At first glance, it seems like the top computers must be gaining considerable ground on the top humans. For instance, it is quite easy to calculate a performance rating for top computers, in their games against humans with FIDE ratings. I did this for 742 games over the past fifteen years, grouped into three-year spans (considering only computers who were one of the top eight engines in the world at the time). Here are the results:

The overall performance rating of computers against humans has increased at roughly 30 Elo points per year. This would suggest that however fast humans are improving, the computers are improving even faster. Thirty Elo points a year faster. However, if you split these same human opponents into two groups, based on a rating cutoff point of 2550, an interesting pattern emerges. Computers are not doing any better against 2550+ players right now, than they were a few years ago.

Although computers have certainly been more successful in 2001-2003 than ever before, that is only due to the fact that they are finally starting to dominate the humans who aren't in the top 200 in the world. As you can see in that graph, computers are doing no better today, against 2550+ opposition, than they were three years ago. Let's look at this event-by-event. First, I'll show you the results against the weaker group, over the past 5-6 years:

Up through mid-2001, a +4 score by a computer, in an event against rated humans, was almost unheard-of. In the two or three years since then, every single computer that played in a tournament against sub-2550 humans, has scored between +4 and +7 against them. Seven straight events, and five different computers, but they all scored at least +4.

On the other hand, you don't see the same level of improvement by computers against the stronger players (in the 2550-2700 range). In fact, you don't see ANY improvement by computers. Let's look at the events since 1998 where computers faced opponents in that rating class:

Computers are becoming more and more dominant against everyone but the top 200 players in the world. That is leading to an overall performance rating for computers that is getting higher and higher. However, the players in the top-200 are holding their ground even against the latest and greatest computers. Perhaps that group will soon shrink down to only the top-100, or the top-50, but not inevitably, and not irreversibly. As you can see from my previous graphics, there is no sign that the top-200 players are losing ground at all against the top computers.

The top 20 humans (the 2700+ crowd) are managing a long string of drawn matches against computers, and the rest of the top-200 is averaging the same 35% to 40% score that they did a few years ago. So, amazing as it may seem, I don't see any evidence that the top computers are suddenly going to take over the chess world. Of course the top computers are improving, mostly through hardware and software upgrades, but somehow the top humans are improving just as fast, in other ways.

In Part I, I mentioned that there were two key questions to answer. Let's review those questions and see what my answers have turned out to be:

Question #1: A decade ago, top grandmasters were undeniably stronger than chess computers. There was a large gap in strength, roughly 300 Elo points. In chess terms, if a top grandmaster had played 100 games against a top computer, the grandmaster would have won the match by a score of about 85-15 (roughly speaking). In the past ten years, computers have certainly reduced the gap. How large is the gap right now, and who is ahead?
Answer #1: There is no measurable gap right now between top computers and top-20 grandmasters. They have been deadlocked for about five years. The remaining top-200 grandmasters, as a group, are slightly weaker than the top computers.

Question #2: Who is improving faster, top grandmasters or chess computers? What can we say about how the situation will be different in one year, or ten years, or fifty years?
Answer #2: There is no measurable difference in how fast either group is improving. There is nothing (yet) to suggest that one or the other will suddenly pull ahead. It may well be that in ten years the top computers and top grandmasters will still be deadlocked.

We are at a unique point in chess history, an unprecedented state of dynamic balance. The top computers have caught up with the top grandmasters, and right now I'm not convinced that the computers will zoom past the grandmasters. Everything depends on whether computers or grandmasters can improve faster, starting today. It may even be that the top humans can figure out how to improve their own anti-computer play, to the point that they will pull ahead again. Perhaps Garry Kasparov can lead the way once more.

There is much more to say about how computers and grandmasters are improving, relative to each other. We'll compare and contrast the various ways that computers and grandmasters are improving, and how we can try to measure those improvements, starting with Part III next week.

Jeff Sonas is a statistical chess analyst who has written dozens of articles since 1999 for several chess websites. He has invented a new rating system and used it to generate 150 years of historical chess ratings for thousands of players. You can explore these ratings on his Chessmetrics website. Jeff is also Chief Architect for Ninaza, providing web-based medical software for clinical trials. Previous articles: Championship Chessmetrics Analysis The Sonas Rating Formula – Better than Elo? How strong are the top chess programs? Have chess computers surpassed the strongest grandmasters?