Dev Builds » 20160916-0615

NCM plays each Stockfish dev build 20,000 times against Stockfish 7. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Results

Host Started (UTC) Duration Base NPS (Avg) Games Wins Losses Draws Elo
ncm-et-3 2018-03-23 09:01 7363:24:17 1987205 2451 624 140 1687 +69.52 +/- 7.47
ncm-et-4 2018-03-23 09:01 7363:24:10 1961809 2385 637 131 1617 +74.85 +/- 7.68
ncm-et-5 2018-03-23 09:01 01:53:48 2019005 761 213 45 503 +77.98 +/- 13.97
ncm-et-6 2018-03-23 09:01 01:53:43 2026391 791 199 29 563 +75.85 +/- 12.48
ncm-et-7 2018-03-23 09:01 01:53:44 2001008 759 204 43 512 +74.83 +/- 13.69
ncm-et-8 2018-03-23 09:01 01:53:46 1986794 754 190 29 535 +75.35 +/- 12.85
ncm-et-9 2018-03-23 09:01 7363:24:32 1990293 2452 635 152 1665 +69.34 +/- 7.59
ncm-et-10 2018-03-23 09:01 7363:24:15 1986914 2467 619 130 1718 +69.79 +/- 7.33
ncm-et-11 2018-03-23 09:01 01:53:42 2004479 758 212 41 505 +79.75 +/- 13.85
ncm-et-12 2018-03-23 09:01 01:53:39 1981512 769 206 43 520 +74.78 +/- 13.57
ncm-et-13 2018-03-23 09:01 7363:24:31 1990819 2456 643 135 1678 +72.92 +/- 7.50
ncm-et-14 2018-03-23 09:01 01:53:50 2019859 768 224 51 493 +79.63 +/- 14.34
ncm-et-15 2018-03-23 09:01 7363:24:02 1986337 2429 636 144 1649 +71.36 +/- 7.62
  20000 5242 1113 13645 +72.77 +/- 2.63

Test Detail

ID Host Started (UTC) Duration Base NPS Games Wins Losses Draws Elo CLI PGN
11293 ncm-et-8 2018-03-23 09:01 01:12:45 2022793 500 124 22 354 +71.89 +/- 15.89 Show
11294 ncm-et-5 2018-03-23 09:01 01:14:02 2014398 500 142 29 329 +79.90 +/- 17.30 Show
11295 ncm-et-14 2018-03-23 09:01 01:12:25 2020184 500 144 31 325 +79.90 +/- 17.54 Show
11296 ncm-et-13 2018-03-23 09:01 01:10:57 2025244 500 145 17 338 +90.97 +/- 16.56 Show
11297 ncm-et-9 2018-03-23 09:01 01:11:37 2024426 500 134 29 337 +74.06 +/- 16.91 Show
11298 ncm-et-10 2018-03-23 09:01 01:11:22 2021818 500 129 35 336 +66.10 +/- 17.07 Show
11299 ncm-et-6 2018-03-23 09:01 01:10:56 2025899 500 129 15 356 +80.63 +/- 15.60 Show
11300 ncm-et-11 2018-03-23 09:01 01:14:41 1990400 500 135 29 336 +74.79 +/- 16.96 Show
11301 ncm-et-7 2018-03-23 09:01 01:14:01 2005531 500 133 27 340 +74.79 +/- 16.71 Show
11302 ncm-et-3 2018-03-23 09:01 01:11:14 2026554 500 108 33 359 +52.51 +/- 15.86 Show
11303 ncm-et-15 2018-03-23 09:01 01:13:22 2019045 500 145 28 327 +82.83 +/- 17.38 Show
11304 ncm-et-12 2018-03-23 09:01 01:11:32 2018883 500 134 28 338 +74.79 +/- 16.84 Show
11305 ncm-et-4 2018-03-23 09:01 01:12:13 2023936 500 139 20 341 +84.31 +/- 16.49 Show
11306 ncm-et-13 2018-03-23 10:14 00:41:05 2025901 288 72 19 197 +64.67 +/- 22.06 Show
11307 ncm-et-6 2018-03-23 10:14 00:41:04 2026882 291 70 14 207 +67.70 +/- 20.79 Show
11308 ncm-et-3 2018-03-23 10:14 00:40:42 2024102 266 70 17 179 +70.16 +/- 23.31 Show
11309 ncm-et-10 2018-03-23 10:14 00:40:23 2021488 282 65 9 208 +69.92 +/- 19.93 Show
11310 ncm-et-9 2018-03-23 10:14 00:40:22 1996019 282 69 14 199 +68.64 +/- 21.34 Show
11311 ncm-et-12 2018-03-23 10:15 00:40:22 1944140 269 72 15 182 +74.75 +/- 22.95 Show
11312 ncm-et-14 2018-03-23 10:15 00:39:41 2019534 268 80 20 168 +79.12 +/- 24.90 Show
11313 ncm-et-4 2018-03-23 10:15 00:39:23 1979486 263 77 15 171 +83.47 +/- 24.13 Show
11314 ncm-et-8 2018-03-23 10:16 00:39:17 1950795 254 66 7 181 +82.20 +/- 21.82 Show
11315 ncm-et-15 2018-03-23 10:16 00:38:36 2018882 259 62 14 183 +65.14 +/- 22.31 Show
11316 ncm-et-5 2018-03-23 10:17 00:38:04 2023611 261 71 16 174 +74.33 +/- 23.72 Show
11317 ncm-et-7 2018-03-23 10:17 00:38:00 1996484 259 71 16 172 +74.92 +/- 23.90 Show
11318 ncm-et-11 2018-03-23 10:18 00:37:18 2018557 258 77 12 169 +89.46 +/- 24.03 Show
128606 ncm-et-9 2019-01-24 00:15 01:14:11 1984722 500 143 34 323 +76.98 +/- 17.69 Show
128607 ncm-et-13 2019-01-24 00:15 01:13:47 1968469 500 114 27 359 +61.08 +/- 15.75 Show
128608 ncm-et-3 2019-01-24 00:15 01:12:16 1965382 500 129 21 350 +76.25 +/- 16.07 Show
128609 ncm-et-10 2019-01-24 00:15 01:12:53 1966314 500 125 26 349 +69.71 +/- 16.24 Show
128610 ncm-et-15 2019-01-24 00:16 01:14:46 1965534 500 118 31 351 +61.08 +/- 16.24 Show
128611 ncm-et-4 2019-01-24 00:16 01:17:46 1893810 500 136 32 332 +73.34 +/- 17.21 Show
128612 ncm-et-3 2019-01-24 01:29 01:14:13 1974860 500 149 28 323 +85.78 +/- 17.57 Show
128613 ncm-et-10 2019-01-24 01:29 01:13:50 1967861 500 138 22 340 +82.10 +/- 16.60 Show
128614 ncm-et-13 2019-01-24 01:30 01:13:16 1972518 500 144 35 321 +76.98 +/- 17.81 Show
128615 ncm-et-9 2019-01-24 01:31 01:14:46 1985212 500 128 38 334 +63.23 +/- 17.22 Show
128616 ncm-et-15 2019-01-24 01:31 01:14:44 1972198 500 132 27 341 +74.06 +/- 16.66 Show
128617 ncm-et-4 2019-01-24 01:35 01:14:59 1939878 500 120 25 355 +66.82 +/- 15.91 Show
128618 ncm-et-3 2019-01-24 02:44 01:12:55 1967874 500 115 30 355 +59.64 +/- 16.01 Show
128619 ncm-et-10 2019-01-24 02:44 01:11:47 1984919 500 114 27 359 +61.08 +/- 15.75 Show
128620 ncm-et-13 2019-01-24 02:45 01:14:58 1974840 500 124 27 349 +68.27 +/- 16.26 Show
128621 ncm-et-9 2019-01-24 02:46 01:13:31 1977486 500 115 30 355 +59.64 +/- 16.01 Show
128622 ncm-et-15 2019-01-24 02:47 01:11:49 1969555 500 126 36 338 +63.23 +/- 16.99 Show
128623 ncm-et-4 2019-01-24 02:51 01:15:15 1960933 500 140 30 330 +77.71 +/- 17.27 Show
128624 ncm-et-10 2019-01-24 03:57 00:28:11 1959081 185 48 11 126 +70.44 +/- 27.57 Show
128625 ncm-et-3 2019-01-24 03:58 00:27:32 1964458 185 53 11 121 +80.28 +/- 28.68 Show
128626 ncm-et-15 2019-01-24 04:00 00:25:08 1972807 170 53 8 109 +94.21 +/- 30.23 Show
128627 ncm-et-13 2019-01-24 04:01 00:25:04 1977941 168 44 10 114 +71.30 +/- 29.05 Show
128628 ncm-et-9 2019-01-24 04:01 00:24:41 1973895 170 46 7 117 +81.15 +/- 28.09 Show
128629 ncm-et-4 2019-01-24 04:07 00:18:44 1972808 122 25 9 88 +45.83 +/- 32.13 Show

Commit

Commit ID 5c58d1f5cb4871595c07e6c2f6931780b5ac05b5
Author Marco Costalba
Date 2016-09-16 06:15:07 UTC
Commit Type Non Functional
Use per-thread counterMoveHistory

Drops a scalability bottleneck due to memory contention
of a single shared table across threads. The effect starts
to be sensible with a high number of threads. Specifically
we have a small regression with 7 threads both at 60 and
180 seconds TC:

10000 @ 60+0.6 th 7
ELO: -2.46 +-3.2 (95%) LOS: 6.5%
Total: 9896 W: 1037 L: 1107 D: 7752

5000 @ 180+0.6 th 7
ELO: -1.95 +-4.1 (95%) LOS: 17.7%
Total: 5000 W: 444 L: 472 D: 4084

We have a regression because counterMoveHistory table is
quite big and it takes time for a single thread to fill it.
Sharing the table yields to a higher fill rate and better
quality of moves and up to 7 threads the benefits of sharing
more then compensate the loss in speed due to contention.
Interestingly even with a 3X longer TC, so with more time
for the single thread to catch up, the improvment is quite
limited and below noise level. It seems we really need much
longer TC to saturate the table.

When we move to high threads number it's another story:

5000 @ 60+0.6 th 22
ELO: 3.49 +-4.3 (95%) LOS: 94.6%
Total: 4880 W: 490 L: 441 D: 3949

2000 @ 60+0.6 th 32
ELO: 8.34 +-6.9 (95%) LOS: 99.1%
Total: 2000 W: 229 L: 181 D: 1590

As expected the speed-up more than compensates the filling
rate, and we expect that with tournament TC, where single
thread is able to saturate the table, the difference will
be even stronger. For instance for TCEC 9 super-final time
control will be 180 minutes + 15 seconds and this scalability
improvement seems definitely the way to go.

So, summarizing:

GOOD:

Measured big improvement in high core scenario

Suitable for TCEC 9 superfinal (big hardware, very long TC)

Consistent and natural patch that extends to counterMoveHistory
what we already do for remaining history tables, that are all per-thread

Non functional change for the common case of a single core

Very simple (just 6 lines modified, no added ones)

BAD:

Small regression (within 2-3 ELO) with few threads and short TC

bench: 5341477

Tests From Startpos

Host Wins Losses Draws Elo
ncm-et-3 4945 1097 220 3628 +62.28 +/- 4.84
ncm-et-4 5055 1094 198 3763 +62.24 +/- 4.68
  10000 2191 418 7391 +62.26 +/- 3.36