Dev Builds » 20160916-0615

NCM plays each Stockfish dev build 20,000 times against Stockfish 7. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Results

Host Started (UTC) Duration Base NPS (Avg) Games Wins Losses Draws Elo
ncm-et-3 2018-03-23 09:01 01:53:40 2025328 766 178 50 538 +58.61 +/- 13.13
ncm-et-4 2018-03-23 09:01 01:53:20 2001711 763 216 35 512 +84.02 +/- 13.63
ncm-et-5 2018-03-23 09:01 01:53:48 2019005 761 213 45 503 +77.98 +/- 13.97
ncm-et-6 2018-03-23 09:01 01:53:43 2026391 791 199 29 563 +75.85 +/- 12.48
ncm-et-7 2018-03-23 09:01 01:53:44 2001008 759 204 43 512 +74.83 +/- 13.69
ncm-et-8 2018-03-23 09:01 01:53:46 1986794 754 190 29 535 +75.35 +/- 12.85
ncm-et-9 2018-03-23 09:01 01:53:42 2010223 782 203 43 536 +72.10 +/- 13.26
ncm-et-10 2018-03-23 09:01 01:53:29 2021653 782 194 44 544 +67.48 +/- 13.07
ncm-et-11 2018-03-23 09:01 01:53:42 2004479 758 212 41 505 +79.75 +/- 13.85
ncm-et-12 2018-03-23 09:01 01:53:39 1981512 769 206 43 520 +74.78 +/- 13.57
ncm-et-13 2018-03-23 09:01 01:53:45 2025573 788 217 36 535 +81.25 +/- 13.25
ncm-et-14 2018-03-23 09:01 01:53:50 2019859 768 224 51 493 +79.63 +/- 14.34
ncm-et-15 2018-03-23 09:01 01:53:41 2018964 759 207 42 510 +76.75 +/- 13.74
  10000 2663 531 6806 +75.23 +/- 3.73

Test Detail

ID Host Started (UTC) Duration Base NPS Games Wins Losses Draws Elo CLI PGN
11293 ncm-et-8 2018-03-23 09:01 01:12:45 2022793 500 124 22 354 +71.89 +/- 15.89 Show
11294 ncm-et-5 2018-03-23 09:01 01:14:02 2014398 500 142 29 329 +79.90 +/- 17.30 Show
11295 ncm-et-14 2018-03-23 09:01 01:12:25 2020184 500 144 31 325 +79.90 +/- 17.54 Show
11296 ncm-et-13 2018-03-23 09:01 01:10:57 2025244 500 145 17 338 +90.97 +/- 16.56 Show
11297 ncm-et-9 2018-03-23 09:01 01:11:37 2024426 500 134 29 337 +74.06 +/- 16.91 Show
11298 ncm-et-10 2018-03-23 09:01 01:11:22 2021818 500 129 35 336 +66.10 +/- 17.07 Show
11299 ncm-et-6 2018-03-23 09:01 01:10:56 2025899 500 129 15 356 +80.63 +/- 15.60 Show
11300 ncm-et-11 2018-03-23 09:01 01:14:41 1990400 500 135 29 336 +74.79 +/- 16.96 Show
11301 ncm-et-7 2018-03-23 09:01 01:14:01 2005531 500 133 27 340 +74.79 +/- 16.71 Show
11302 ncm-et-3 2018-03-23 09:01 01:11:14 2026554 500 108 33 359 +52.51 +/- 15.86 Show
11303 ncm-et-15 2018-03-23 09:01 01:13:22 2019045 500 145 28 327 +82.83 +/- 17.38 Show
11304 ncm-et-12 2018-03-23 09:01 01:11:32 2018883 500 134 28 338 +74.79 +/- 16.84 Show
11305 ncm-et-4 2018-03-23 09:01 01:12:13 2023936 500 139 20 341 +84.31 +/- 16.49 Show
11306 ncm-et-13 2018-03-23 10:14 00:41:05 2025901 288 72 19 197 +64.67 +/- 22.06 Show
11307 ncm-et-6 2018-03-23 10:14 00:41:04 2026882 291 70 14 207 +67.70 +/- 20.79 Show
11308 ncm-et-3 2018-03-23 10:14 00:40:42 2024102 266 70 17 179 +70.16 +/- 23.31 Show
11309 ncm-et-10 2018-03-23 10:14 00:40:23 2021488 282 65 9 208 +69.92 +/- 19.93 Show
11310 ncm-et-9 2018-03-23 10:14 00:40:22 1996019 282 69 14 199 +68.64 +/- 21.34 Show
11311 ncm-et-12 2018-03-23 10:15 00:40:22 1944140 269 72 15 182 +74.75 +/- 22.95 Show
11312 ncm-et-14 2018-03-23 10:15 00:39:41 2019534 268 80 20 168 +79.12 +/- 24.90 Show
11313 ncm-et-4 2018-03-23 10:15 00:39:23 1979486 263 77 15 171 +83.47 +/- 24.13 Show
11314 ncm-et-8 2018-03-23 10:16 00:39:17 1950795 254 66 7 181 +82.20 +/- 21.82 Show
11315 ncm-et-15 2018-03-23 10:16 00:38:36 2018882 259 62 14 183 +65.14 +/- 22.31 Show
11316 ncm-et-5 2018-03-23 10:17 00:38:04 2023611 261 71 16 174 +74.33 +/- 23.72 Show
11317 ncm-et-7 2018-03-23 10:17 00:38:00 1996484 259 71 16 172 +74.92 +/- 23.90 Show
11318 ncm-et-11 2018-03-23 10:18 00:37:18 2018557 258 77 12 169 +89.46 +/- 24.03 Show

Commit

Commit ID 5c58d1f5cb4871595c07e6c2f6931780b5ac05b5
Author Marco Costalba
Date 2016-09-16 06:15:07 UTC
Commit Type Non Functional
Use per-thread counterMoveHistory

Drops a scalability bottleneck due to memory contention
of a single shared table across threads. The effect starts
to be sensible with a high number of threads. Specifically
we have a small regression with 7 threads both at 60 and
180 seconds TC:

10000 @ 60+0.6 th 7
ELO: -2.46 +-3.2 (95%) LOS: 6.5%
Total: 9896 W: 1037 L: 1107 D: 7752

5000 @ 180+0.6 th 7
ELO: -1.95 +-4.1 (95%) LOS: 17.7%
Total: 5000 W: 444 L: 472 D: 4084

We have a regression because counterMoveHistory table is
quite big and it takes time for a single thread to fill it.
Sharing the table yields to a higher fill rate and better
quality of moves and up to 7 threads the benefits of sharing
more then compensate the loss in speed due to contention.
Interestingly even with a 3X longer TC, so with more time
for the single thread to catch up, the improvment is quite
limited and below noise level. It seems we really need much
longer TC to saturate the table.

When we move to high threads number it's another story:

5000 @ 60+0.6 th 22
ELO: 3.49 +-4.3 (95%) LOS: 94.6%
Total: 4880 W: 490 L: 441 D: 3949

2000 @ 60+0.6 th 32
ELO: 8.34 +-6.9 (95%) LOS: 99.1%
Total: 2000 W: 229 L: 181 D: 1590

As expected the speed-up more than compensates the filling
rate, and we expect that with tournament TC, where single
thread is able to saturate the table, the difference will
be even stronger. For instance for TCEC 9 super-final time
control will be 180 minutes + 15 seconds and this scalability
improvement seems definitely the way to go.

So, summarizing:

GOOD:

Measured big improvement in high core scenario

Suitable for TCEC 9 superfinal (big hardware, very long TC)

Consistent and natural patch that extends to counterMoveHistory
what we already do for remaining history tables, that are all per-thread

Non functional change for the common case of a single core

Very simple (just 6 lines modified, no added ones)

BAD:

Small regression (within 2-3 ELO) with few threads and short TC

bench: 5341477

Tests From Startpos

Host Wins Losses Draws Elo
ncm-et-3 4945 1097 220 3628 +62.28 +/- 4.84
ncm-et-4 5055 1094 198 3763 +62.24 +/- 4.68
  10000 2191 418 7391 +62.26 +/- 3.36