Go Players Should Not Trust AI Win Rate

Quentin Rendu /December 7, 2023

How to cite this article:
Rendu, Q. (2023). Go Players Should Not Trust AI Win Rate. Journal of Go Studies, 17(2), 61-88. doi: 10.62578/476299

Abstract
      The advent of artificial intelligence (AI) has transformed the landscape of various strategic games, including Go. In 2016, the AI-powered engine AlphaGo defeated one of the world’s strongest players. Since then, Go engines have routinely been used by amateur and professional Go players to analyse theirgames. In the early stages of AI analysis, Go players relied solely on the AI win rate, the only available indicator. However, the AI win rate does not accurately reflect the win rate of human Go players and might be misleading.
      Katago, first released in 2019, is the first engine to provide score predictions in addition to win rates. While it is now possible to evaluate board positions with a score, it remains unclear how this score translates into human win rates. In this work, a large database of online and professional games is analysed to extract the win rate of a human player based on their strength and the stage of the game. As expected, the human win rate is significantly lower than the AI win rate, even for 9dan professional players. A general formula is provided to compute the win rate based on player strength and move number. This feature offers new insights into the relative importance of mistakes and can assist players in making improved decisions during games.

Keywords: Go, Baduk, Weiqi, AI, Katago, Win rate, Statistics