NCM plays each Stockfish dev build 20,000 times against Stockfish 14. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.
Host | Duration | Avg Base NPS | Games | WLD | Standard Elo | Ptnml(0-2) | Gamepair Elo |
---|---|---|---|---|---|---|---|
ncm-dbt-01 | 02:57:48 | 1092380 | 1000 | 419 65 516 | +128.55 +/- 9.58 | 0 11 137 339 13 | +289.36 +/- 29.25 |
ncm-dbt-02 | 02:55:29 | 1204364 | 1000 | 439 72 489 | +133.74 +/- 9.5 | 0 11 124 352 13 | +306.84 +/- 30.78 |
ncm-dbt-03 | 02:55:00 | 1199614 | 1000 | 450 83 467 | +133.74 +/- 10.13 | 0 13 128 338 21 | +295.94 +/- 30.28 |
ncm-dbt-05 | 02:57:18 | 1197707 | 1000 | 426 73 501 | +128.16 +/- 9.71 | 0 12 137 337 14 | +286.77 +/- 29.25 |
ncm-dbt-06 | 02:58:15 | 1190849 | 1000 | 434 80 486 | +128.55 +/- 10.31 | 1 14 134 332 19 | +282.94 +/- 29.57 |
5000 | 2168 373 2459 | +130.54 +/- 4.41 | 1 61 660 1698 80 | +292.23 +/- 13.27 |
ID | Host | Base NPS | Games | WLD | Standard Elo | Ptnml(0-2) | Gamepair Elo | CLI | PGN | ||
---|---|---|---|---|---|---|---|---|---|---|---|
367403 | ncm-dbt-05 | 1198236 | 500 | 208 34 258 | +126.17 +/- 13.77 | 0 5 74 163 8 | +277.93 +/- 39.92 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 849641617 \ -pgnout ncm-dbt-20240509-0629-010.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367402 | ncm-dbt-01 | 1079051 | 500 | 213 31 256 | +132.54 +/- 13.65 | 0 5 66 171 8 | +298.62 +/- 42.41 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 1100151868 \ -pgnout ncm-dbt-20240509-0629-009.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367401 | ncm-dbt-06 | 1189006 | 500 | 220 36 244 | +134.15 +/- 14.5 | 0 10 54 178 8 | +304.07 +/- 46.54 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 3985567056 \ -pgnout ncm-dbt-20240509-0629-008.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367400 | ncm-dbt-02 | 1199105 | 500 | 228 31 241 | +144.71 +/- 13.08 | 0 3 56 182 9 | +339.63 +/- 46.2 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 1567274438 \ -pgnout ncm-dbt-20240509-0629-007.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367399 | ncm-dbt-03 | 1196029 | 500 | 224 32 244 | +140.62 +/- 14.53 | 0 8 53 178 11 | +318.25 +/- 47.23 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 45243219 \ -pgnout ncm-dbt-20240509-0629-006.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367398 | ncm-dbt-05 | 1197179 | 500 | 218 39 243 | +130.14 +/- 13.7 | 0 7 63 174 6 | +295.94 +/- 43.42 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 3642665874 \ -pgnout ncm-dbt-20240509-0629-005.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367397 | ncm-dbt-06 | 1192693 | 500 | 214 44 242 | +123.02 +/- 14.65 | 1 4 80 154 11 | +263.42 +/- 38.27 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 1284090504 \ -pgnout ncm-dbt-20240509-0629-004.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367396 | ncm-dbt-01 | 1105710 | 500 | 206 34 260 | +124.6 +/- 13.45 | 0 6 71 168 5 | +280.42 +/- 40.83 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 3422619668 \ -pgnout ncm-dbt-20240509-0629-003.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367395 | ncm-dbt-02 | 1209623 | 500 | 211 41 248 | +123.02 +/- 13.65 | 0 8 68 170 4 | +277.93 +/- 41.74 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 3610874260 \ -pgnout ncm-dbt-20240509-0629-002.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
|||||||||||
367394 | ncm-dbt-03 | 1203199 | 500 | 226 51 223 | +126.97 +/- 14.11 | 0 5 75 160 10 | +275.45 +/- 39.63 | ↓ | |||
cutechess-cli \ -rounds 259 \ -games 2 \ -concurrency 9 \ -srand 2999582957 \ -pgnout ncm-dbt-20240509-0629-001.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=64 \ option.Threads=2 \ -engine \ name=20240509-0629 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:3bdfa0fb4a837f51f142cc1e862837c6f9167796 \ -engine \ name=sf14 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:14 |
Commit ID | 3bdfa0fb4a837f51f142cc1e862837c6f9167796 |
---|---|
Author | FauziAkram |
Date | 2024-05-09 06:29:33 UTC |
Depth dependent statscore based reductions
Test a modification of Fawzi's PR #5223, against that PR.
parameters locally tuned with nevergrad4sf.
passed STC:
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 1047424 W: 271478 L: 269649 D: 506297
Ptnml(0-2): 3851, 124543, 265290, 125982, 4046
https://tests.stockfishchess.org/tests/view/663b0889ca93dad645f7c58c
passed LTC:
LLR: 2.96 (-2.94,2.94) <0.50,2.50>
Total: 796236 W: 201712 L: 199825 D: 394699
Ptnml(0-2): 361, 88381, 218778, 90206, 392
https://tests.stockfishchess.org/tests/view/663be6adca93dad645f7f509
https://github.com/official-stockfish/Stockfish/pull/5228
Bench: 3346224
|