Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Prevent negative checkmate scores from being lower than EvaluationConstants.MinEval #1063

Merged
merged 3 commits into from
Sep 27, 2024

Conversation

eduherminio
Copy link
Member

@eduherminio eduherminio commented Sep 27, 2024

Distance MinEval from -CheckMateBaseEvaluation enough so that there's no possible interference between them at TT level.
Now I reaaaally want to change EvaluationConstants.CheckmateDepthFactor to 1, but now it's not the time.

Logs seen when saving an item to a TT.

RecordHash: for position "6k1/4b1p1/8/4P3/8/q7/6r1/2K5 w - - 0 1", with raw score -29865, saving -30085 from TT
RecordHash: for position "6k1/4b1p1/8/4P3/6r1/1q6/8/1K6 w - - 0 1", with raw score -29865, saving -30085 from TT

Logs seen when retrieving that item from TT, ending up with a score < EvaluationConstants.MinEval (-30001).

ProbeHash: for position "6k1/4b1p1/8/4P3/8/q7/6r1/2K5 w - - 0 1", with raw score -30085, returning -30005 from TT. Alpha -29921^, Beta -29920, Type "Alpha", Recalculated score -30005
Returning -30005 from TT at depth 0, ply 8
Full search in first visited move returned -30005

This which eventually causes some NegaMax calls to use that value and return it, which ends up in the situation where other shallower NegaMax instances have a bestScore that never bets beaten in the fail-soft version of alphabeta, eventually causing an empty PV and a crash due to a8a8 move

Logs produced using b8f23b9, from bugfix/no-pruning-negative-checkmate-score-debugging (#1062)


Code-flow explanation, dumbed for future me:

  • The negative checkmate scores issue was already there since.. forever
    If bestScore is EvaluationContants.MinEval, the double invocations to TranspositionTableExtensions.RecalculateMateScores on saving an entry and on retrieving it lead the resulting score to be lower than EvaluationContants.MinEval.
  • While adopting fail soft, I directly jumped into the nested implementation:
              if (score > bestScore)
              {
                  bestScore = score;
    
                  if (score >= beta) { /* ... */} }
                  if (score > alpha) { /* ... */} }
              }
    vs the option of checking alpha and beta outside of the if (score > bestScore conditional.
    This implies that if the returned score is ever lower than bestScore, which gets initialized to EvaluationContants.MinEval, there's never a fail low node that beats alpha, that is there's no move beating the 'no move', so there's no PV populated.
  • This leads to the issue in Aspiration windows implementation of the IDDFS:`
                      while (true)
                      {
                          bestScore = NegaMax(depth: depth, ply: 0, alpha, beta);
                          window += window >> 1;   // window / 2
    
                          if (alpha >= bestScore)     // Fail low
                          {
                              alpha = Math.Max(bestScore - window, EvaluationConstants.MinEval);
                              beta = (alpha + beta) >> 1;  // (alpha + beta) / 2
                          }
                          else if (beta <= bestScore)     // Fail high
                              beta = Math.Min(bestScore + window, EvaluationConstants.MaxEval);
                          else
                              break;
                      }
    In our case, bestScore is lower than EvaluationConstants.MinEval but alpha is restricted to [minEval, ...), so alpha always beats bestScore. Given that, we end up in the situation described in https://github.com/🐛 Avoid aspiration windows after a checkmate has been detected in the search #1057 (but not fixed): an infinite Aspiration windows loop where stuff overflows due to the window variable growing indefinitely, variables start to overflow and a cancellation with still no PV ends up happening
Lynx.Engine|Eval (-30001) (depth 8, nodes 84) outside of aspiration window, new window [-30001, -29995] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 168) outside of aspiration window, new window [-30001, -29998] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 252) outside of aspiration window, new window [-30001, -30000] 
Lynx.Engine|Eval (-29980) (depth 8, nodes 309) outside of aspiration window, new window [-30001, -29917] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 399) outside of aspiration window, new window [-30001, -29959] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 483) outside of aspiration window, new window [-30001, -29980] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 567) outside of aspiration window, new window [-30001, -29991] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 651) outside of aspiration window, new window [-30001, -29996] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 735) outside of aspiration window, new window [-30001, -29999] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 819) outside of aspiration window, new window [-30001, -30000] 
Lynx.Engine|Eval (-29980) (depth 8, nodes 854) outside of aspiration window, new window [-30001, -28914] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 940) outside of aspiration window, new window [-30001, -29458] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1026) outside of aspiration window, new window [-30001, -29730] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1112) outside of aspiration window, new window [-30001, -29866] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1198) outside of aspiration window, new window [-30001, -29934] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1282) outside of aspiration window, new window [-30001, -29968] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1366) outside of aspiration window, new window [-30001, -29985] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1450) outside of aspiration window, new window [-30001, -29993] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1534) outside of aspiration window, new window [-30001, -29997] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 1618) outside of aspiration window, new window [-30001, -29999]
....
Lynx.Engine|Eval (-29980) (depth 8, nodes 3052) outside of aspiration window, new window [-30001, 30001] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3138) outside of aspiration window, new window [-30001, 0] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3224) outside of aspiration window, new window [-30001, -15001] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3310) outside of aspiration window, new window [-30001, -22501] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3396) outside of aspiration window, new window [-30001, -26251] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3482) outside of aspiration window, new window [-30001, -28126] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3568) outside of aspiration window, new window [-30001, -29064] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3654) outside of aspiration window, new window [-30001, -29533] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3740) outside of aspiration window, new window [-30001, -29767] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3826) outside of aspiration window, new window [1967549553, 983759893] 
Lynx.Engine|Eval (-29960) (depth 8, nodes 3828) outside of aspiration window, new window [-30001, 491864946] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 3914) outside of aspiration window, new window [-30001, 245917472] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 4000) outside of aspiration window, new window [1271841875, 758879673] 
Lynx.Engine|Eval (-29960) (depth 8, nodes 4002) outside of aspiration window, new window [1907777854, -814154885] 
Lynx.Engine|Eval (-29920) (depth 8, nodes 4010) outside of aspiration window, new window [-30001, -407092443] 
Lynx.Engine|Eval (-29960) (depth 8, nodes 4018) outside of aspiration window, new window [-30001, -2145113894] 
Lynx.Engine|Eval (-29960) (depth 8, nodes 4026) outside of aspiration window, new window [-30001, 30001] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 4112) outside of aspiration window, new window [-30001, 0] 
Lynx.Engine|Eval (-30001) (depth 8, nodes 4198) outside of aspiration window, new window [1870919157, 935459578] 
Lynx.Engine|Eval (-29960) (depth 8, nodes 4200) outside of aspiration window, new window [-30001, 467714788]
...
Search cancellation requested after 625ms (depth 8, nodes 963346), best move will be returned
[Lynx]	info depth 7 seldepth 13 multipv 1 score cp -30001 nodes 963346 nps 592828 time 625 hashfull 78 pv 
[Lynx]	bestmove a8a8

Test  | bugfix/tt-min-eval-vs-checkmated-scores
Elo   | -0.67 +- 1.90 (95%)
SPRT  | 8.0+0.08s Threads=1 Hash=32MB
LLR   | 2.93 (-2.25, 2.89) [-5.00, 0.00]
Games | 49608: +13007 -13102 =23499
Penta | [964, 5769, 11415, 5710, 946]
https://openbench.lynx-chess.com/test/788/

Another 60k games vs the GOAT without illegal moves
8+0.08, no win adjudication,

Score of Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64 vs Stockfish 17: 527 - 28391 - 1722  [0.045] 30640
...      Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64 playing White: 325 - 13392 - 1603  [0.074] 15320
...      Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64 playing Black: 202 - 14999 - 119  [0.017] 15320
...      White vs Black: 15324 - 13594 - 1722  [0.528] 30640
Elo difference: -529.5 +/- 7.7, LOS: 0.0 %, DrawRatio: 5.6 %
SPRT: llr 0 (0.0%), lbound -inf, ubound inf

Player: Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64
   "Draw by 3-fold repetition": 343
   "Draw by adjudication": 806
   "Draw by fifty moves rule": 108
   "Draw by insufficient mating material": 130
   "Draw by stalemate": 6
   "Draw by timeout": 329
   "Loss: Black loses on time": 25
   "Loss: Black mates": 13294
   "Loss: White loses on time": 98
   "Loss: White mates": 14974
   "Win: Black loses on time": 314
   "Win: White loses on time": 202
   "Win: White mates": 11
Player: Stockfish 17
   "Draw by 3-fold repetition": 343
   "Draw by adjudication": 806
   "Draw by fifty moves rule": 108
   "Draw by insufficient mating material": 130
   "Draw by stalemate": 6
   "Draw by timeout": 329
   "Loss: Black loses on time": 314
   "Loss: White loses on time": 202
   "Loss: White mates": 11
   "Win: Black loses on time": 25
   "Win: Black mates": 13294
   "Win: White loses on time": 98
   "Win: White mates": 14974

Score of Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64 vs Stockfish 17: 191 - 29197 - 1522  [0.031] 30910
...      Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64 playing White: 110 - 13867 - 1478  [0.055] 15455
...      Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64 playing Black: 81 - 15330 - 44  [0.007] 15455
...      White vs Black: 15440 - 13948 - 1522  [0.524] 30910
Elo difference: -599.2 +/- 8.6, LOS: 0.0 %, DrawRatio: 4.9 %
SPRT: llr 0 (0.0%), lbound -inf, ubound inf

Player: Lynx-bugfix-no-pruning-negative-checkmate-score-debugging-4110-win-x64
   "Draw by 3-fold repetition": 360
   "Draw by adjudication": 799
   "Draw by fifty moves rule": 122
   "Draw by insufficient mating material": 107
   "Draw by stalemate": 9
   "Draw by timeout": 125
   "Loss: Black loses on time": 3
   "Loss: Black mates": 13849
   "Loss: White loses on time": 18
   "Loss: White mates": 15327
   "Win: Black loses on time": 99
   "Win: White loses on time": 81
   "Win: White mates": 11
Player: Stockfish 17
   "Draw by 3-fold repetition": 360
   "Draw by adjudication": 799
   "Draw by fifty moves rule": 122
   "Draw by insufficient mating material": 107
   "Draw by stalemate": 9
   "Draw by timeout": 125
   "Loss: Black loses on time": 99
   "Loss: White loses on time": 81
   "Loss: White mates": 11
   "Win: Black loses on time": 3
   "Win: Black mates": 13849
   "Win: White loses on time": 18
   "Win: White mates": 15327

…re's no possible interference between them at TT level
@eduherminio eduherminio changed the title 🐛 Prevent negative checkmate scores from being lower than EvaluationConstants.MinEval 🐛 Prevent negative checkmate scores from being lower than EvaluationConstants.MinEval Sep 27, 2024
@eduherminio eduherminio marked this pull request as ready for review September 27, 2024 20:59
@eduherminio eduherminio merged commit 154491f into main Sep 27, 2024
27 checks passed
@eduherminio eduherminio deleted the bugfix/tt-min-eval-vs-checkmated-scores branch September 27, 2024 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant