For the list of engines and downloadable files, see CCRL - Games (computerchess.org.uk/ccrl).
When was the last time you saw a research paper based on a study of chess960? There must have been others, but this is the first one I can remember seeing. Titled Machine Learning Algorithms to Predict Chess960 Result and Develop Opening Themes (arxiv.org; 'submitted on 29 Oct 2023'), it lists its authors as:-
Shreyan Deo, DPS Vasant Kunj, Delhi, India • Nishchal Dwivedi, Department of Basic Science and Humanities, SVKM’s NMIMS Mukesh Patel School of Technology Management & Engineering, Mumbai, India
The 'Abstract' offers a useful overview. In its entirety it says,
This work focuses on the analysis of Chess 960, also known as Fischer Random Chess, a variant of traditional chess where the starting positions of the pieces are randomized. The study aims to predict the game outcome using machine learning techniques and develop an opening theme for each starting position.
The first part of the analysis utilizes machine learning models to predict the game result based on certain moves in each position. The methodology involves segregating raw data from PGN files [NB: CCRL was the source] into usable formats and creating datasets comprising approximately 500 games for each starting position. Three machine learning algorithms -- KNN Clustering, Random Forest, and Gradient Boosted Trees -- have been used to predict the game outcome.
To establish an opening theme, the board is divided into five regions: center, white kingside, white queenside, black kingside, and black queenside. The data from games played by top engines in all 960 positions is used to track the movement of pieces in the opening. By analysing the change in the number of pieces in each region at specific moves, the report predicts the region towards which the game is developing. These models provide valuable insights into predicting game outcomes and understanding the opening theme in Chess 960.
Keywords: Chess 960 [Chess960], Fischer Random Chess, machine learning, game outcome prediction, opening theme, KNN Clustering, Neural Networks, Gradient Boosted Trees.
Note: Trying to see just by looking at the snapshot of an evolved game how accurately we can predict who will win
The 16-page paper is interesting for many reasons. I extracted the section headers to create a table of contents:-
1. Introduction
2. Methodology
2.1. Segregation of Data
2.2. Visualising Chess Positons as Numbers
2.3. Predicting the Outcome
2.4. Developing an Opening Theme
2. Modelling [should be '3.'?]
3.1. KNN Clustering
3.2. Random Forest
3.3. Gradient Boosted Trees
4. Result
4.1. Outcome Prediction
4.2. Theme Analysis
5. Discussion
5.1. Discussing the Accuracy of the Predictions
5.1.1. Comparing the Different Data Sets
5.1.2. Comparing the Different Machine Learning Models
5.1.3. Disparity in the Accuracies of Certain Starting Positions
5.1.4. Possible Reasons for Low Accuracy Rate
5.2. Discussing the Theme Analysis and Its Importance
6. Conclusion
7. Future Developments & Limitations
8. References [eight total]
9. Appendix
The section '9. Appendix' starts,
This section traces the opening theme of each individual starting position. The heading provided is in the form X_Y, where X represents the region in which White pieces tend to develop, and Y represents the region in which Black pieces develop.
This is very useful in individually finding out how each specific starting position should be played. For example, the position BBNNQRKR should be played by the players such that White attacks the Black Queenside whereas Black attacks the White Kingside.
Why choose SP032 BBNNQRKR as an example? Because it's the study's first position in the first group of positions, all of which are under the header 'Black Q Side_White K Side'. A useful diagram ('Figure 1: Regions of the Chess Board') from the study helps to explain how it sees the different sections of the board.
Unfortunately, the explanation ('should be played by the players such that...') of BBNNQRKR is dubious:-
The position BBNNQRKR clearly shows that the two Bishops ('BB') on the Queenside are attacking the castling pieces ('RKR') bunched together on the Kingside. That indicates how both sides will proceed. How should White attack the Black Queenside? With the Knights ('NN')?
At this point it's worth remembering that at the start of each game both players have identical piece placements. The difference is that one side moves first and thereby has a natural initiative.
The traditional start position (SP518 RNBQKBNR), is grouped under 'Black Q Side_Centre', meaning:-
Here it's worth remembering that White has a choice of first moves (1.e4, 1.d4, etc.) with different strategies for developing the pieces. The other 959 start positions also offer a choice of strategies. I can't remember playing many (correspondence) chess960 games where White has a first move that is clearly better (NB: 'SP864 BBQRKRNN' & 'SP868 QBBRKRNN' are not clear). Is it possible to group the subsequent developments of different first moves for one start position under a single header, e.g. 'White attacks... _ Black attacks...' ?
The rest of the paper is equally unconvincing, although there are large portions that I didn't understand. One key illustration ('Figure 3') is incomprehensible because the column headers are clipped to be identical.
An important discussion starts with '5.1. Discussing the Accuracy of the Predictions'. It says,
The results of the machine learning models were quite surprising yet also logical. All the predictions were only [around] 40% accurate, which is only slightly better than randomly guessing an outcome and getting it right, the probability of which is 33.33% (as there are only 3 outcomes possible).
Another section, '5.1.4. Possible Reasons for Low Accuracy Rate', offers several 'possible reasons', but ignores the most obvious. The methodology of the study might well be seriously flawed.
One more idea that could be easily tested is to look at the chess018 start positions (SPs) on CCRL. Which SP is statistically best? Which is worst? What are the most popular first moves for each SP? Sounds like another post is taking form.
The following chart lists the 18 SPs in the chess018 family. The first three columns are from my own database of SPs. The last three columns are from the CCRL (see the sidebar for a link).
The column 'Dist' ('Distance') is a metric I whipped up when I first started investigating chess960. It was explained in the following posts:-
Some day I hope to find a use for 'Dist', e.g. a correlation with some other observation. Today is not that day.
The CCRL numbers for chess018 should be compared to the same numbers across all 960 SPs. At first glance, nothing unusual appears. Somewhat curious is the one SP where the score for White is less than 50% -- the traditional start position. I imagine it has something to do with the engines not using a book.
The fourth [rating] list was based on chess960 games. Here are the top 25 engines from that list.
I received a question -- @nicbentulan: Is Leela on this list? (twitter.com) -- against the Twitter anchor for the post. A quick look at the list reveals that Leela is on the most recent list at no.42, rated 3008:-
no.42 • Lc0 0.29.0 CPU_744706 • 3008
That rating is nearly 1000 points lower than no.1 'Stockfish 15.1', currently rated 4005. The Lc0 rating looks dubious, given that Leela is currently one of the top three engines in the world. How has it done in TCEC FRC competitions? Following is a list of posts from this blog:-
Why is Leela so far down on the CCRL rating list? I'll leave that for the CCRL to answer.
I see that there are four CCRL rating lists. Shown below are the top five engines from three of the lists.
Why didn't I show all four lists? Because the fourth list was based on chess960 games. Here are the top 25 engines from that list.
The most striking feature of the list is that the top engine, Stockfish 15.1, is rated over 4000. This is more than 250 points higher than the top rating shown on the '3600 Barrier' chart. In fact the top four engines on the CCRL FRC list are higher then the top engine on the earlier list. I couldn't determine why this is.
[NB: The domain given here (ccrl.chessdom.com) is not the same as the domain in the right sidebar (computerchess.org.uk), although the two sites appear to be identical. The ratings are based on the same database of games used to compute the CCRL 'Opening statistics'.]
Stockfish evaluation at depth ~40 for all the starting positions
This is also known as the 'Sesse' resource and I gave its URL in the post. The Molas study concluded,
Stockfish evaluations don’t predict actual winning rates for each variation
This didn't surprise me. If you consider that each start position (SP) leads to a mega-zillion possible games and that Sesse reduces each SP to a single two-digit number, much more surprising would be to find a meaningful correlation between an SP's W-L-D percentages and its Sesse value.
I discussed the Sesse numbers once before in A Stockfish Experiment (February 2019). That post mentioned another discussion, What's the Most Unbalanced Chess960 Position? (chess.com; Mike Klein; March 2018 / February 2020). Fun Master (FM) Mike observed,
Let's now take the most extreme case the other way -- the position where Sesse claims White enjoys the most sizable advantage. The lineup BBNNRKRQ delivers a whopping +0.57 plus for the first move. The advantaged is so marked that some chess960 events may even jettison this arrangement as a possible option (a total of four positions are +0.50 or better for White, but none are as lopsided as this one).
That position, also known as 'SP080 BBNNRKRQ', has received some notoriety thanks to Sesse, so I decided to investigate further. I downloaded the SP080 file from the CCRL (see link in the right sidebar), loaded it into SCID, and discovered that it contained 554 games. SCID gave me percentages for White's first moves, which I copied into the following chart.
There are 11 first moves for White listed in the top block of the chart. I then expanded the first two of those moves -- 1.g3 (65.7% overall score for White) and 1.Nd3 (59.7%) -- into the second and third blocks of the chart to see how Black has responded to those moves.
You might be wondering why I said there were 554 games in the file, but the SCID extract counts only 519 games. SCID was designed to handle the traditional start position (SP518 RNBQKBNR) and knows nothing about chess960 castling rules. SP080 allows 1.O-O on the first move, which SCID rejects. The 35 missing games (554 minus 519) are games that started 1.O-O. When I'm using SCID for a chess960 correspondence game, I have a technique to account for this anomaly, but I won't go into details here.
Similarly, the charts for 1.g3 and 1.Nd3 show '[end]' as one of the first moves for Black. These are games where Black played 1...O-O on the first move. The corresponding percentage scores are among the worst for Black, showing once again that early castling is a risky strategy.
If I were playing SP080 in a correspondence game, I would analyze both 1.g3 and 1.Nd3. A promising continuation after 1.g3 is 1...c5, which the score '43.9%' says, 'Favors Black'. Of course, I would have to look at White's second moves in this variation, where one move will appear to be superior to the others. And so on and so on.
To be useful, the SCID tool needs to be handled intelligently. I recently blundered into a wrong evaluation that I doumented in The CCRL Is Unreliable (Not!) (December 2021). I'm hopeful that some day a tool will appear that rivals SCID functionality *and* that understands chess960 castling. For now, I make do with the software I have.
For a look at two more SPs where evaluations have shifted with experience, see SP864 - BBQRKRNN and SP868 - QBBRKRNN, which are both attachments to this blog. One lesson I've learned from playing chess960 for almost 15 years : nothing is fixed in stone.
Some start positions are forced wins for White
Remembering that I wrote this more than 10 years ago, at a time when I wasn't absolutely 100% sure that such unfair positions didn't exist, my standard response to the statement was, 'Which positions are forced wins? Please provide a specific example'. I never received a single example. Ten years later I can say with more confidence -- although still not 'absolutely 100% sure' -- that while some positions are difficult for Black to play, none of the 960 positions is lost before a single move is made.
In January a new study titled, Analyzing Chess960 Data | Alex Molas | Towards Data Science (towardsdatascience.com), appeared. Its subtitle announced,
Using more than 14 million chess960 games to find if there’s a variation that's better than the others.
There is considerable knowledge presented in the study and I don't pretend to understand all of it. I might well need several posts to unravel its subtleties, so I'll start by summarizing its references; in the following discussion, '>>>' means a direct quote from 'Analyzing Chess960 Data'.
>>> 'The original post was published here...'
[NB: I'll come back to this reference later; see '(A)' below. First I need to point out that there's an important issue with terminology. When chess players use the term 'variation', they mean a sequence of play arising from a specific position; e.g. 'In this position I had two variations and I had to work out which variation was better for me.' • In the Molas study, I'm convinced that the word 'variation' refers to one of the well-defined 960 start positions that are legal for chess960. I read the subtitle of the towardsdatascience.com article as 'to find if there’s a *start position* that’s better than the others' and the title of the amolas.dev post as saying 'Discovering the best chess960 *start position*'. I won't repeat this caveat each time, but it's important and helps to understand the discussion.]
>>> 'Ryan Wiley wrote this blog post where he analyzes some data from lichess..'
>>> 'There’s also this repo with the statistics for 4.5 millions games (~4500 games per variation)...'
[NB: There's an issue with the word 'variant' here, but it's not as important as the previous 'NB'. Chess960 purists will know what I'm talking about.]
>>> 'In this spreadsheet there’s the Stockfish evaluation at depth ~40 for all the starting positions...'
>>> 'There’s also this database with Chess960 games between different computer engines. However, I’m currently only interested in analyzing human games, so I’ll not put a lot of attention to this type of games...'
>>> 'Lichess -- the greatest chess platform out -- maintains a database with all the games that have been played in their platform...'
>>> 'To do the analysis, I downloaded ALL the available Chess960 data (up until 31–12–2022). For all the games played I extracted the variation, the players Elo and the final result...'
>>> 'The scripts and notebooks to donwload [sic] and process the data are available on this repo...'
At this point the article launches into 'Mathematical framework; 'Bayesian A/B testing; [...]'. This, of course, is the essence of the study and I won't go any further in this current post. Let's get back to '(A)', where there's another key reference.
>>> 'This post got some attention in Reddit...'
I could end the post here, but I need to make an admittedly subjective observation. There are two example of bias in the above references.
The first bias is 'I’m currently only interested in analyzing human games'. Huge caveat here. In my not-so-humble opinion, the CCRL is the best source of chess960 opening theory. Period. Full stop. The CCRL engines are rated at least 1000 points higher than most human players on Lichess. The engines don't make simple tactical errors and they calculate deeper into every position than any human can. If there is an unfair chess960 start position, the engines will find it, just like they find errors in most games played between humans.
I can understand ignoring the engines because humans grapple with different challenges in chess960 openings, but the purpose of the study was 'to find if there’s a *start position* that’s better than the others'. Ignoring the experience of the best players on the planet is severely limiting.
The second bias is 'Lichess -- the greatest chess platform out'. The main alternative here is Chess.com. Why ignore games played on the world's largest chess platform? Maybe there's a good reason, but I can't think of one. On a personal note, last year I investigated which of the two sites would be better to continue my own chess960 correspondence play. I determined that Chess.com was more serious about eliminating human players who cheat by using engines in games with other humans. Since my goal was playing no-engine games, I went with Chess.com. How much of the Lichess data involves concealed engine use?
Biases notwithstanding, the Molas study is an important step in evaluating the fairness of all 960 positions in chess960/FRC. I'm looking forward to understanding it in more depth.
This month I started a new eight-game (four different start positions) event and decided to take another look at the CCRL data. One of the games, SP777 QRKBBNRN, is pictured below.
SP777 QRKBBNRN
For the game where I had white, I chose 1.Nhg3 as the first move, and the game continued...
1.Nhg3 [x89] b5 [x2] 2.e4 [x12]
...reaching the position in the diagram. The numbers in brackets ('[]') are the number of games in the CCRL file for SP777, out of 378 games total. After Black's first move, only two CCRL games remained, but the number increased after White's second move, thanks to a simple transposition of White's first and second moves.
The first block of text in the image, statistics calculated by SCID, shows Black's moves after 2.e4. The numbers are terrible for White -- in the 12 games, White scored only 25%. When I first saw the stats, I told myself, 'This line is unplayable!', but couldn't see any other reason why it shouldn't be played. It's the most logical move in the position.
The second block of text shows some basic info about the 12 games, listed in chronological order. The last two columns are the most interesting. The WLD scores total +2-8=2 (25%), but the last column shows the 'Length' (number of moves) for each game. Of the 12 games, only four lasted longer than 15 moves.
The first game, Hiarcs - Movei (played 2006), ended '1-0' after Black's 6th move, when the opponent's are still developing their forces. The last game, Stockfish - Dragon (played this year), ended after White's 15th move with the comment 'Black wins by adjudication', although the position is at best unclear.
My conclusion? The CCRL statistics are completely misleading and can't be relied on. My own SP777 game hasn't yet reached the 10th move, so I can't say anything else, because it's still in progress.
***
Later: Re 'Of the 12 games, only four lasted longer than 15 moves', after I wrote the post, I realized I had made a serious error. SCID doesn't know anything about chess960. When it encounters an illegal move, it stops processing the game. For chess960 games, most castling moves look like errors, so SCID stops. Thus the low move counts.
For the first three games in my list, the real plycounts were 99, 511, and 122, where the number of 'moves' in the game (more accurately called 'move pairs', i.e. a move for White plus a move for Black) would be half that. This reminded me of an equally serious error I made on my main blog in the first few months of chess960 blogging, when I miscalculated CCRL statistics:-
My conclusion now? (1) I'm the one that's unreliable, not the CCRL; (2) I shouldn't blog on Christmas day, the date of this post; (3) I should be careful when I find a simple answer to a complicated question.
Re the original observation, 'The numbers are terrible for White -- in the 12 games, White scored only 25%', I'll have to look at it again. The game that provoked the analysis has reached the 17th move, so it is still too early to comment.
Recently, the Norwegian "supercomputer" Sesse analyzed all 960 variations using Stockfish 9. At a depth of 39-40 ply for each position, which took about two to three hours each, its findings were published.
I downloaded the referenced spreadsheet and compared it with the data currently displayed on the CCRL (see link in the right sidebar). The results are shown below.
The first column shows the start position, followed by the Sesse evaluation in centipawns, followed by the CCRL overall percentage score for White and the precentage of draws. For example, the top row says that BBNNRKRQ was evaluated by Sesse at 0.57 (a little more than a half-Pawn), with a CCRL result of 51.0% in White's favor and 15.3% of games ending in a draw.
The chart gives the top-25 positions flagged by Sesse, down to a cutoff of 0.40. I could have added the numeric ID for each start position and will do so if I ever come back to the data.
At first glance, there doesn't seem to be a high correlation between the Sesse results and the CCRL results. The first position where there is agreement between the two sources is QBBRKRNN. I once discussed this position in A Difficult SP for Black (April 2013). By coincidence, I'm currently playing the position in a pair of correspondence games and hope to have more to say about it in the future.
As for the Sesse results, they confirm that engines just don't evaluate chess960 start positions particularly well. Long term positional considerations are not the engines' strong point.
Under the assumption that every chess960 position has exactly the same first-move advantage, by natural variation I get results just as extreme as the ones our blogger has compiled. So perhaps some positions have just been lucky for White so far, and others unlucky, with no inherent bias. At a minimum, if these are the most conclusive stats available, we have to say there is so far no statistical evidence that some positions favor White more than others.
In other words, the CCRL results match the distribution one would expect from the number of games in the CCRL sample, assuming a 55%-45% theoretical advantage for White. I asked Ichabod, the chess960 expert and professional statistician last seen on this blog in A Better Pawn Method, if he agreed with the post on the Arimaa forum and he confirmed its methodology. Then I asked him, 'How big would the samples have to be to reduce the extremes to their theoretical minimum?' He answered,
It's not a question of theoretical minimum. The question is, are the results you are seeing more extreme than you would expect with random chance? If you aren't, then there isn't statistical evidence of an effect.
To clarify, you need to think about how much of an advantage you want to detect. What you're seeing is that at your current sample size you can't detect an advantage of 12% because of the random noise. What advantage do you want to detect? 5%? 1%? From that you could back calculate a necessary sample size from the multinomial win/loss/draw distribution.
Last year, on my main blog, I posted a series on Practical Evaluation, where I learned that the value of the first move in traditional chess is a half-tempo, which is worth 0.2 times the value of a Pawn. In the last post in the series, I learned that A Pawn Equals 200 Rating Points, which gives White a theoretical advantage of 56%-44% based on the half-tempo. This is very close to the observed advantage for White over millions of games.
Given that all of the start positions in chess960 confer a half-tempo advantage on White, does that mean White always has an advantage of 56%-44%? Or perhaps the half-tempo advantage isn't equivalent to 0.2 times a Pawn for all 960 start positions. I suspect the latter is true, but how will we ever find out, given that we need so many games with each start position to provide a valid sample.
I asked Ichabod, 'How many games would I have to play in another start position to know that the new W%-B% is significantly different?' He answered,
Here we get into the issue of the two different kinds of significance: statistical significance and practical significance. Statistical significance is going to determine what sample size you need to detect a given difference. Practical significance is going to determine what difference you want to detect. Say we had a bazillion games for each position, and we could show that in some positions White had an advantage of 0.000001 pawns. No one would care. We would have statistical significance but we wouldn't have practical significance.
On the other hand, let's say in certain positions we could show with statistical significance that white had a full Pawn advantage. Then people would care, and would think that position is flawed. We would have both practical and statistical significance. Now, somewhere between 0.000001 pawns and a full Pawn is a minimum advantage that would be considered a practically significant difference between the standard position and a given chess960 position.
Determining that minimum advantage is not a statistics question, it's a chess question. That is, you have to determine what fraction of a Pawn advantage is a practically significant advantage.
Here's an idea for killing a large amount of time: Run an engine (any chess960-enabled engine) on all 960 start positions (SPs). Record the value of the top-10 first moves for each SP. Analyze the results. Can any information be derived from the observed value of the first moves?
The TCEC Grand champion Stockfish, modification 260614, won another prestigious computer chess competition – the Fischer Random Chess (FRC) tournament, organized as a TCEC Season 6 Special Event. [...] The reigning champion Stockfish was convincing with 25/28 points, leaving the runner-up Houdini 4 full 3 points behind. Critter 1.6a took the bronze, collecting 17,5 points.
TCEC is, of course, the Thoresen Chess Engines Competition. In May, Chessdom posted an Interview with Martin Thoresen – organizer and director of TCEC, where we learn that
Some people refer to TCEC as the "unofficial Computer World Championship". The organization ICGA is hosting the official World Championship, but they have lost a lot of interest over the years – in particular because none of the strongest engines are participating there and they play very few games.
and that
I [Thoresen] was a part of the CCRL rating list prior to starting TCEC. I ran a lot of test games for them, but it really didn’t interest me as much after a while – I was more interested in organizing tournaments and broadcasting it for others to watch
I downloaded the PGN file containing the games from the event and will look at it in another post. These games will make a valuable addition to my own copy of the Chess Jungle's database; see Chess960 Database, Part II for more info.
The seven games that started two weeks ago have all advanced to around moves 8-12. All of them are still in theory, meaning that I can find games on Chesslab.com with the same position I'm playing. Of the seven games, three are still in basic theory, where hundreds of games have already been played to the position in my game. In one of those three games, my opponent is taking a 'long think' to decide between two theoretical moves of equal importance that branch into completely different paths.
Of the other four games, three are in positions where I find 10-20 previously played games. In all three of those games, one variation has been played more than any other, and the effort required on my part is to understand why that variation is so popular and where it will eventually lead. Previous experience tells me that all of those games will eventually reach a position where several different moves have been tried, each move having a handful (less than five) of representative games. That's the point where I have to start thinking for myself.
The last of the seven games has already reached the 'handful' stage. That, of course, is where the chess starts to become interesting because I'm really on my own.
In contrast to those seven SP518 games, I just started two chess960 correspondence games, one with White and one with Black. Both games are on the first move and I'm already on my own. I know from experience that the CCRL database (see the link in the right sidebar) is unreliable for opening ideas, since engines just don't understand the opening. I also know that I won't find more than a few dozen games elsewhere using the same start position, many of those between players who have a different idea about opening objectives than I have.
I know that preparing for correspondence chess is not the same as preparing for crossboard chess. In fact, preparing openings for crossboard chess is even more time consuming and the work is never finished. In a world where time is a limited quantity for all of us, that pretty much sums up why I prefer playing chess960.
HarryO and I set out to play the SP in the comments to a post on his blog: Non-Random Chess 960 Trial Game 6: SP408. He chose to play 1.d4, the same move played in 64 of the 112 CCRL games, and which had garnered a success rate of 75% for White. The resulting position is shown in the following diagram.
SP408 RBQNBNKR after 1.d4
Now it was my move. I noted in my first comment,
From CCRL I'm seeing 1.d4 with a WLD score of +44-12=8. The two most important variations are 1...c5 with +22-9=6 and 1...Ng6 with +20-2=0. Those are terrible stats for Black.
Later I added,
After 1.d4 c5 2.d5, the most popular move is 2...c4. It's not a bug. The engine is trying to prevent c2-c4, which creates a strong center for White. Note that White has also moved the same Pawn twice, but the second move is very strong because it limits the movement of the Black Knights. I don't think 1...c5 is playable.
On top of that, I don't like 1...Ng6. It commits the Knight to a less than optimal square and leaves White a free hand in the center. I appreciate that it prepares ...O-O and guards the weak e-Pawn, but neither of these objectives is a priority.
I finally decided to play 1...d5, a move which had not been tried in any of the 112 CCRL games. HarryO played the critical move 2.c4, against which I had prepared 2...Nde6. The move depends on the correctness of the tactical sequence 3.cxd5 Nxd4. We played through move 16, and although Black never achieved full equality, he was never in real danger of losing in the opening.
Looking again at the comments we made while making the moves, I am impressed by our general discoveries about chess960. After the game, HarryO expanded the anchor post to highlight some of the unusual variations that might have been played.
Our next game started with the twin of SP408: Non-Random Chess 960 Trial Game 7: SP749. It has the same sequence of pieces, but in reverse order -- only the castling considerations are different. The first two moves for each side mirrored the ideas discovered in SP408. White varied first, playing the equivalent of 3.e3. This put less pressure on Black and when we finally abandoned the trial on move 15, Black had achieved equality and was perhaps even somewhat better.
Getting back to the initial question -- Were the lopsided CCRL results 'because of a problem with the SP or a problem with the engines?' -- I'm convinced that it's a problem with the engines. A discussion of why they go wrong would be a good topic for a future post.
We started off with a test of Non-random Fischer Random, where the players actively choose the start position (SP) themselves, rather than rely on a random process. We played two games, taking a different color in each game. Links to HarryO's posts, where the moves and running commentary can be found, are marked HO; links to my posts are marked MW.
Although the non-random trials worked well enough, we decided to tackle several specific SPs that appear to push the chess960 concept to its limits. Our methodology involves playing enough moves to determine that Black is not overwhelmed in the opening and emerges with a playable game.
Toward the end of last year, I wrote a post that flagged other problematic SPs: Waving a Yellow Flag. HarryO proposed tackling those and we were off again.
Looks like I still need to cover SP408 and SP749. To date we haven't found any SPs that are hopeless for Black, but that doesn't mean they are easy to play.
The following table shows the results. For example, Bishop starting on the a-file scores 51.55% for White overall and draws in 20.51% of the games.
Although the numbers look like statistical noise, a few patterns might be significant. According to the table, Bishops starting in the corner win less for White than when starting on other squares. Bishops on b/g-files draw less. Knights in the corner win more. Queens in the corner draw less.
Why only 'a good idea at the time'? I expected to see something more convincing.
The second pair of twins exhibiting widely varying success rates for White and Black were SP301 QNRBKRBN and SP376 NBRKBRNQ. The position SP301 had an average success rate for White (%W) of 47.0%, while SP376 had %W of 62.9. That means SP301 was more successful for Black, and SP376 more successful for White.
The following chart shows the two SPs next to the results collected from the CCRL's engine vs. engine data (once again, see the sidebar for a link to the CCRL data). For example, the data for SP301 says that the most popular move, 1.Ng3, was played in 89 games and had a 43.2% success rate for White (%W again). The second most popular move was only played in 14 games. I haven't listed the other first moves given by the CCRL for that SP, all of which were seen in less than 10 games. The second block of stats for SP301 shows all of the responses to 1.Ng3. The move 1...f6 was played in 47 of the 89 games and was largely responsible for the low %W.
The data for SP376 tells a different story. While 1.Nb3 (the logical counterpart of SP301's 1.Ng3) was again the most popular first move, the response 1...c6 (the counterpart to SP301's 1...f6) is further down the list. The top response to 1.Nb3 was 1...Nb6, a move whose SP301 counterpart 1...Ng6 is nowhere to be found. What's going on here?
As luck would have it, I've already discussed this pair of twins in another post: Watch out for ****KRBN. In SP301, after 1.Ng3, White is threatening a dangerous attack on g7. The most common opening sequence on CCRL was 1.Ng3 (x89 moves) f6 (x47) 2.f4 (x34) Ng6 (x19). Now there were three moves (total x22 due to transpositions), all played about the same number of times, and all with %W less than 50%. The worst of the three moves was 3.Nf5 with a score of +1-6=0, i.e. a disaster for White.
In SP376, after 1.Nb3 there is no attack on b7 because of the option to castle ...O-O-O. The most common opening sequence was 1.Nb3 (x88) Nb6 (x38) 2.c4 (x21), and now there were two moves more popular than 2...O-O-O (where White won all games). The first move was 2...g6, where the score was +1-7=0; the second was 2...Na4, with a score of +5-1=0. These scores, both disasters for different sides, make no sense to me. Perhaps they can best be explained by the small number of games played.
In both SPs, it appears that the side playing for the attack is 'playing to lose', as the old chess saying goes. I would love to investigate this further, but I'm afraid I've run out of time.
SP222 NQRKNBBR has [an average success rate for White (%W)] of only 43.3%, while its twin, SP644 RBBNKRQN, has %W of 59.8%.
Is this a statistical fluke or is there an underlying reason? I noted,
Four of the five twins with the largest difference in %W have one SP in the pair where castling is possible on the first move. This indicates that the ability to castle quickly is an important defensive strategy in a difficult position.
After posting those observations, I noticed that the four pairs of twins didn't support my conclusion. Where castling was possible on the first move, White had a significantly better score in two of the pairs, while Black was better in the other two. I decided to take a closer look at the twins from my example, SP222 and SP644.
The following table shows some basic information for the two SPs. The column on the right shows the most frequent first moves for each SP along with each move's score for White. For example, in SP222 the move 1.Nb3 was played 35 times and scored 42.8% for White; in SP644 the equivalent move 1.Ng3 was played 36 times and scored 48.6% for White. The two Knight moves appear to affect their respective SPs in the same way.
I doubt that the difference between SP222's 1.Nb3 score of 42.8% and SP644's 1.Ng3 score of 48.6% is statistically significant. A more promising investigation would be SP222's 1.c4 score of 35.7% and SP644's 1.f4 score of 61.2%. Both first moves involve pushing the Pawn in front of a Rook. It so happens that the chosen Pawn is relevant to the position of the castled King on that side.
In SP222, the move 1.c4 weakens an eventual O-O-O; in SP644, the move 1.f4 makes space for the Rook after an eventual O-O. Is that the reason for the difference in scores for White? More investigation is required to answer that question.
Two of those five SPs were twins, positions with the pieces in reverse order relative to each other. Twins have similar characteristics for early piece development and tactics -- only different castling options distinguish them -- and early problems for Black in one twin could easily occur in the other twin. I decided it would be useful to identify other pairs of twins with high success rates for White, so I returned to the data described in the 'Yellow Flag' post to find more.
The following table shows the five twins with the highest average success rate for White (%W AVG). SP408 and its twin head the list, since they were the twins that I had already identified (the lower numbered SP is always listed in the left column). The traditional start position, SP518, and its twin are given as a reference point. The next five entries in the table are the twins with the lowest average success rate for White. If there are any start characteristics that favor Black, they might be hidden in these SPs.
At the bottom of the table are five twins with the highest difference in %W. For example, the first entry SP222 NQRKNBBR has a success rate (%W) of only 43.3%, while its twin, SP644 RBBNKRQN, has %W of 59.8%. The difference between the two values for %W is -16.5. It is of no importance that the lower numbered SP or the higher is more successful; only the absolute value of the difference is important.
Four of the five twins with the largest difference in %W have one SP in the pair where castling is possible on the first move. This indicates that the ability to castle quickly is an important defensive strategy in a difficult position. The only exception to the possibility of early castling is the third pair of twins, SP424 RBNQBNKR and SP733 RKNBQNBR. Why do these twins have such a large difference in success rate for White?
There are plenty of avenues for further investigation here, and I hope that HarryO and I will find the time to tackle a few of them. If so, I'll report on our discoveries in future posts.
CCRL means "Computer Chess Rating Lists". We are a club of people inspired by watching computers play chess.
While I'm not at all 'inspired by watching computers play chess', I do accept that the CCRL findings are the single most important collection of chess960 games in existence today. I ended the 'Revisited' post by concluding that the CCRL sample at that time was probably too small to make generalizations. The club now boasts of having played 115.900 games of chess960, or an average of more than 115 games per start position, or 3-4 times more games than at the time of my first post.
How do the current CCRL statistics compare with that first analysis? Here is a table similar to my earlier table. As before, it shows the number of games played (#); the percentage of games won by White, won by Black, and drawn (%W, %B, %D); and the overall score for White (%S).
| SP | FEN | # | %W | %B | %D | %S |
|---|---|---|---|---|---|---|
| Highest Overall Score White | ||||||
| 408 | RBQNBNKR | 112 | 57.1 | 29.5 | 13.4 | 63.8 |
| 868 | QBBRKRNN | 120 | 54.2 | 26.6 | 19.2 | 63.8 |
| 749 | RKNBNQBR | 116 | 52.6 | 25.8 | 21.6 | 63.4 |
| 730 | RKNQBBNR | 114 | 55.3 | 28.9 | 15.8 | 63.2 |
| 376 | NBRKBRNQ | 140 | 56.4 | 30.6 | 12.9 | 62.9 |
| Highest Overall Score Black | ||||||
| 465 | BRNBNKRQ | 106 | 34.0 | 50.0 | 16.0 | 42.0 |
| 786 | BRQKNBRN | 114 | 30.7 | 47.3 | 21.9 | 41.7 |
| 888 | RBQKBRNN | 118 | 27.1 | 44.1 | 28.8 | 41.5 |
| 499 | BRQNKNRB | 118 | 29.7 | 49.1 | 21.2 | 40.3 |
| 067 | BNNRKQRB | 116 | 28.5 | 49.1 | 22.4 | 39.7 |
| Traditional Start | ||||||
| 518 | RNBQKBNR | 135 | 44.4 | 34.8 | 20.7 | 54.8 |
| Traditional Start (K&Q switched) | ||||||
| 534 | RNBKQBNR | 84 | 34.5 | 35.7 | 29.8 | 49.4 |
The first observation is that the 'the overall score for White (%S)' lies in a narrower range than before: 64%-42% now vs. 76%-32% earlier. The second observation is that of the top five percentages for White, only one start position is on both lists (SP868 QBBRKRNN); of the bottom five percentages for White (i.e. top five for Black), the list is completely different. The third observation is that the traditional start position (SP518 RNBQKBNR) is now in line with experience.
On the earlier list, I examined the top position for White (SP024 NBQNBRKR) in A Followup, an Error, and an Insight, and surmised (with the help of another chess960 enthusiast) that 'the dangerous start positions share the common feature of several Pawns unprotected and these can be easily attacked by the Queen and Bishop'. That is clearly the case for SP868, where the 'QBB' on the a/b/c-files rake the 'RNN' f/g/h-files. For the record, SP024 is now no.12 on the list, with a 60.9% success rate for White rather than the 76.2% in the earlier table. It's worth remembering that the CCRL requires both 'Book learning' and 'Position learning' to be 'Off for all engines'. This means that the engines don't improve by playing more games.
Now here's a big, yellow WARNING FLAG: the no.1 and the no.3 entries in the current table (SP408 RBQNBNKR & SP749 RKNBNQBR) are twins -- they have the same sequence of pieces, but in reverse order -- only the castling considerations are different. These two positions, plus the repeat position SP868, need to be examined more closely. I would also like to identify the other top (and bottom) twins to determine how random their CCRL results are. I'll follow-up these actions in future posts.
Black must have been very surprised when his opponent replied 2.Nh5, attacking the Pawn on g7, a Pawn which cannot be defended. Even worse, the Knight move threatens smothered mate. Black answered 2...f6, along with the comment, 'Ugly starting position and I bungled it.' It's like giving odds of Pawn and move.
SP301 QNRBKRBN
What should Black have played in response to 1.Ng3? One idea is 1...f6, opening the diagonal for the Bg8. If then 2.Nh5, Black has 2...g6, and if White continues 3.Ng7+, the Knight is trapped after 3...Kf7. Another idea is 1...b6, with a counterattack on White's g2. The obvious move is 1...g6, preventing the White Knight from going to either f5 or h5, but I don't like this move because it does nothing for Black's development and takes the natural square from the Knight on h8.
A little thought convinced me that any position starting ****KRBN is susceptible to the same tactic. How many such positions are there? Since the remaining White Bishop must be placed on one of the two light squares, when the other three pieces can then be placed on any empty square, there are 2 x 3 x 2 = 12 positions. The same tactic is possible for any twin of those 12 positions, i.e. the positions starting NBRK****, although these have the additional possibility of castling 1...O-O-O.
How do the CCRL engines (see the right sidebar for a link) handle SP301? Of the 82 games currently archived there, White opened 1.Ng3 in 63 of them. The following table shows how many times Black responded with various moves:-
1...f6 x 34
1...f5 x 16
1...g6 x 5
1...b6 x 3
1...e6 x 3
1...e5 x 2
The move 1...f6 is the overwhelming favorite, followed by the surprising 1...f5. It's no surprise, however, that the unfortunate 1...Ng6 was not selected by any of the engines.
