Dev Builds » 20230606-1917

Use this dev build

NCM plays each Stockfish dev build 20,000 times against Stockfish 15. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Summary

Host	Duration	Avg Base NPS	Games	WLD	Standard Elo	Ptnml(0-2)	Gamepair Elo
ncm-dbt-01	06:54:36	583716	4014	1313 688 2013	+54.54 ± 5.11	5 193 990 810 9	+111.14 ± 10.78
ncm-dbt-02	06:54:07	585626	4000	1302 715 1983	+51.36 ± 5.1	3 208 993 791 5	+104.68 ± 10.77
ncm-dbt-03	06:53:29	585337	4000	1312 719 1969	+51.89 ± 5.13	2 210 991 787 10	+104.68 ± 10.78
ncm-dbt-04	06:54:17	570597	4000	1306 741 1953	+49.41 ± 5.03	1 208 1022 763 6	+99.95 ± 10.59
ncm-dbt-05	06:54:41	582966	3986	1300 736 1950	+49.49 ± 5.09	3 207 1014 761 8	+100.13 ± 10.64
			20000	6533 3599 9868	+51.34 ± 2.28	14 1026 5010 3912 38	+104.11 ± 4.79

Test Detail

ID	Host	Base NPS	Games	WLD	Standard Elo	Ptnml(0-2)	Gamepair Elo	PGN
408680	ncm-dbt-01	582569	14	6 2 6	+102.04 ± 70.73	0 0 3 4 0	+225.59 ± 263.0	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2519007916 \ -pgnout ncm-dbt-20230606-1917-041.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408679	ncm-dbt-05	585126	486	161 91 234	+50.39 ± 14.45	1 23 124 95 0	+104.56 ± 30.47	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2364268115 \ -pgnout ncm-dbt-20230606-1917-040.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408678	ncm-dbt-02	587367	500	163 92 245	+49.67 ± 15.03	1 27 125 94 3	+98.44 ± 30.45	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3955651748 \ -pgnout ncm-dbt-20230606-1917-039.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408677	ncm-dbt-04	569310	500	160 101 239	+41.19 ± 14.43	0 30 133 85 2	+80.63 ± 29.38	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2944169536 \ -pgnout ncm-dbt-20230606-1917-038.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408676	ncm-dbt-03	587707	500	174 100 226	+51.8 ± 14.08	0 23 132 93 2	+102.97 ± 29.38	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 4129775311 \ -pgnout ncm-dbt-20230606-1917-037.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408675	ncm-dbt-01	585928	500	173 80 247	+65.38 ± 14.0	0 19 120 110 1	+134.15 ± 31.01	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2379173597 \ -pgnout ncm-dbt-20230606-1917-036.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408674	ncm-dbt-05	582527	500	158 90 252	+47.54 ± 13.85	0 24 135 90 1	+95.44 ± 28.99	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3485844824 \ -pgnout ncm-dbt-20230606-1917-035.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408673	ncm-dbt-02	587792	500	173 82 245	+63.95 ± 13.36	0 16 127 107 0	+132.54 ± 29.87	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1437749485 \ -pgnout ncm-dbt-20230606-1917-034.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408672	ncm-dbt-03	585169	500	161 95 244	+46.13 ± 14.72	0 31 123 95 1	+92.46 ± 30.75	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1816419674 \ -pgnout ncm-dbt-20230606-1917-033.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408671	ncm-dbt-04	570869	500	169 98 233	+49.67 ± 14.64	1 27 122 100 0	+102.97 ± 30.86	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1682120214 \ -pgnout ncm-dbt-20230606-1917-032.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408670	ncm-dbt-01	584664	500	150 86 264	+44.72 ± 15.15	1 31 123 93 2	+89.48 ± 30.76	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1055453113 \ -pgnout ncm-dbt-20230606-1917-031.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408669	ncm-dbt-05	580862	500	157 97 246	+41.89 ± 14.99	0 35 121 93 1	+83.57 ± 31.04	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1113248099 \ -pgnout ncm-dbt-20230606-1917-030.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408668	ncm-dbt-02	584495	500	168 77 255	+63.95 ± 14.37	0 23 113 114 0	+132.54 ± 32.13	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1431301355 \ -pgnout ncm-dbt-20230606-1917-029.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408667	ncm-dbt-03	586646	500	167 82 251	+59.64 ± 14.59	1 21 122 104 2	+121.46 ± 30.78	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1795762379 \ -pgnout ncm-dbt-20230606-1917-028.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408666	ncm-dbt-04	570829	500	164 94 242	+48.96 ± 13.93	0 25 130 95 0	+99.95 ± 29.7	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3158750424 \ -pgnout ncm-dbt-20230606-1917-027.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408665	ncm-dbt-01	581361	500	160 86 254	+51.8 ± 13.94	0 24 128 98 0	+106.01 ± 29.96	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3007273184 \ -pgnout ncm-dbt-20230606-1917-026.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408664	ncm-dbt-05	583196	500	162 91 247	+49.67 ± 13.97	1 22 132 95 0	+102.97 ± 29.38	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1232342508 \ -pgnout ncm-dbt-20230606-1917-025.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408663	ncm-dbt-04	571391	500	166 86 248	+56.07 ± 14.43	0 23 127 97 3	+110.6 ± 30.08	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2815700882 \ -pgnout ncm-dbt-20230606-1917-024.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408662	ncm-dbt-02	585464	500	159 98 243	+42.6 ± 14.52	1 28 131 89 1	+86.52 ± 29.64	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2879977972 \ -pgnout ncm-dbt-20230606-1917-023.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408661	ncm-dbt-03	585632	500	167 87 246	+56.07 ± 14.43	0 25 121 103 1	+113.68 ± 30.97	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2544394164 \ -pgnout ncm-dbt-20230606-1917-022.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408660	ncm-dbt-01	583656	500	166 83 251	+58.21 ± 14.39	1 21 123 104 1	+119.89 ± 30.63	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2167901900 \ -pgnout ncm-dbt-20230606-1917-021.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408659	ncm-dbt-05	581194	500	164 89 247	+52.51 ± 14.25	1 23 126 100 0	+109.07 ± 30.24	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2048717189 \ -pgnout ncm-dbt-20230606-1917-020.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408658	ncm-dbt-04	570148	500	161 102 237	+41.19 ± 13.89	0 28 135 87 0	+83.57 ± 29.08	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2840001035 \ -pgnout ncm-dbt-20230606-1917-019.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408657	ncm-dbt-02	585970	500	159 89 252	+48.96 ± 15.13	1 30 118 100 1	+99.95 ± 31.44	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 451647770 \ -pgnout ncm-dbt-20230606-1917-018.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408656	ncm-dbt-03	585211	500	147 81 272	+46.13 ± 14.18	0 28 128 94 0	+93.95 ± 30.03	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2936670794 \ -pgnout ncm-dbt-20230606-1917-017.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408655	ncm-dbt-01	581527	500	168 88 244	+56.07 ± 14.15	0 23 125 101 1	+113.68 ± 30.37	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 4078288912 \ -pgnout ncm-dbt-20230606-1917-016.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408654	ncm-dbt-05	586816	500	164 92 244	+50.38 ± 14.14	0 25 129 95 1	+101.46 ± 29.84	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2071514070 \ -pgnout ncm-dbt-20230606-1917-015.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408653	ncm-dbt-02	584411	500	160 85 255	+52.51 ± 14.66	0 29 117 104 0	+107.54 ± 31.57	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1552352097 \ -pgnout ncm-dbt-20230606-1917-014.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408652	ncm-dbt-04	568275	500	163 83 254	+56.07 ± 14.01	0 22 127 100 1	+113.68 ± 30.06	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 556951745 \ -pgnout ncm-dbt-20230606-1917-013.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408651	ncm-dbt-01	585042	500	159 91 250	+47.55 ± 15.18	1 29 124 93 3	+93.95 ± 30.6	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3170622222 \ -pgnout ncm-dbt-20230606-1917-012.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408650	ncm-dbt-03	584706	500	162 88 250	+51.8 ± 14.49	0 28 120 102 0	+106.01 ± 31.14	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2304131809 \ -pgnout ncm-dbt-20230606-1917-011.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408649	ncm-dbt-05	582903	500	169 92 239	+53.93 ± 14.32	0 24 127 97 2	+107.54 ± 30.1	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3779028240 \ -pgnout ncm-dbt-20230606-1917-010.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408648	ncm-dbt-02	582736	500	167 88 245	+55.36 ± 14.26	0 25 121 104 0	+113.68 ± 30.97	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2968975614 \ -pgnout ncm-dbt-20230606-1917-009.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408647	ncm-dbt-04	570629	500	161 85 254	+53.22 ± 14.01	0 24 126 100 0	+109.07 ± 30.24	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 73396061 \ -pgnout ncm-dbt-20230606-1917-008.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408646	ncm-dbt-01	585169	500	166 87 247	+55.36 ± 14.53	2 20 126 101 1	+115.23 ± 30.2	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 47474267 \ -pgnout ncm-dbt-20230606-1917-007.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408645	ncm-dbt-03	581402	500	169 88 243	+56.78 ± 14.86	1 25 117 106 1	+116.78 ± 31.56	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 1005291587 \ -pgnout ncm-dbt-20230606-1917-006.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408644	ncm-dbt-05	581111	500	165 94 241	+49.67 ± 15.16	0 31 120 96 3	+96.94 ± 31.16	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 2330202142 \ -pgnout ncm-dbt-20230606-1917-005.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408643	ncm-dbt-02	586774	500	153 104 243	+34.16 ± 13.72	0 30 141 79 0	+68.99 ± 28.31	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 138340377 \ -pgnout ncm-dbt-20230606-1917-004.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408642	ncm-dbt-04	573325	500	162 92 246	+48.96 ± 14.47	0 29 122 99 0	+99.95 ± 30.87	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 4271988354 \ -pgnout ncm-dbt-20230606-1917-003.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408641	ncm-dbt-01	583531	500	165 85 250	+56.07 ± 14.43	0 26 118 106 0	+115.23 ± 31.41	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3401136093 \ -pgnout ncm-dbt-20230606-1917-002.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15
408640	ncm-dbt-03	586223	500	165 98 237	+46.84 ± 14.75	0 29 128 90 3	+90.97 ± 30.05	↓
cutechess-cli \ -rounds 266 \ -games 2 \ -concurrency 16 \ -srand 3984400037 \ -pgnout ncm-dbt-20230606-1917-001.pgn \ -openings \ file=UHO_4060_v2.epd \ format=epd \ order=random \ -repeat \ -resign \ movecount=3 \ score=600 \ -draw \ movenumber=34 \ movecount=8 \ score=5 \ -each \ tc=30+0.3 \ timemargin=10000 \ proto=uci \ option.Hash=128 \ option.Threads=8 \ -engine \ name=20230606-1917 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=dev_build:373359b44d0947cce2628a9a8c9b432a458615a8 \ -engine \ name=sf15 \ cmd=docker \ arg=run \ arg=-i \ arg=--rm \ arg=--entrypoint=/engine \ arg=stockfish:15

Commit

Commit ID	373359b44d0947cce2628a9a8c9b432a458615a8
Author	Linmiao Xu
Date	2023-06-06 19:17:36 UTC
Update default net to nn-0dd1cebea573.nnue Created by retraining an earlier epoch of the experiment leading to the first SFNNv6 net on a more-randomized version of the nn-e1fb1ade4432.nnue dataset mixed with unfiltered T80 apr2023 data. Trained using early-fen-skipping 28 and max-epoch 960. The trainer settings and epochs used in the 5-step training sequence leading here were: 1. train from scratch for 400 epochs, lambda 1.0, constant LR 9.75e-4, T79T77-filter-v6-dd.min.binpack 2. retrain ep379, max-epoch 800, end-lambda 0.75, T60T70wIsRightFarseerT60T74T75T76.binpack 3. retrain ep679, max-epoch 800, end-lambda 0.75, skip 28, nn-e1fb1ade4432 dataset 4. retrain ep799, max-epoch 800, end-lambda 0.7, skip 28, nn-e1fb1ade4432 dataset 5. retrain ep439, max-epoch 960, end-lambda 0.7, skip 28, shuffled nn-e1fb1ade4432 + T80 apr2023 This net was epoch 559 of the final (step 5) retraining: ```bash python3 easy_train.py \ --experiment-name L1-1536-Re4-leela96-dfrc99-T60novdec-v2-T80juntonovjanfebT79aprmayT78jantosepT77dec-v6dd-T80apr-shuffled-sk28 \ --training-dataset /data/leela96-dfrc99-T60novdec-v2-T80juntonovjanfebT79aprmayT78jantosepT77dec-v6dd-T80apr.binpack \ --nnue-pytorch-branch linrock/nnue-pytorch/misc-fixes-L1-1536 \ --early-fen-skipping 28 \ --start-lambda 1.0 \ --end-lambda 0.7 \ --max_epoch 960 \ --start-from-engine-test-net False \ --start-from-model /data/L1-1536-Re3-nn-epoch439.nnue \ --engine-test-branch linrock/Stockfish/L1-1536 \ --lr 4.375e-4 \ --gamma 0.995 \ --tui False \ --seed $RANDOM \ --gpus "0," ``` During data preparation, most binpacks were unminimized by removing positions with score 32002 (`VALUE_NONE`). This makes the tradeoff of increasing dataset filesize on disk to increase the randomness of positions in interleaved datasets. The code used for unminimizing is at: https://github.com/linrock/Stockfish/tree/tools-unminify For preparing the dataset used in this experiment: ```bash python3 interleave_binpacks.py \ leela96-filt-v2.binpack \ dfrc99-16tb7p-eval-filt-v2.binpack \ filt-v6-dd-min/test60-novdec2021-12tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd-min/test80-aug2022-16tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd-min/test80-sep2022-16tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd-min/test80-jun2022-16tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd/test80-jul2022-16tb7p-filter-v6-dd.binpack \ filt-v6-dd/test80-oct2022-16tb7p-filter-v6-dd.binpack \ filt-v6-dd/test80-nov2022-16tb7p-filter-v6-dd.binpack \ filt-v6-dd-min/test80-jan2023-3of3-16tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd-min/test80-feb2023-16tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd/test79-apr2022-16tb7p-filter-v6-dd.binpack \ filt-v6-dd/test79-may2022-16tb7p-filter-v6-dd.binpack \ filt-v6-dd-min/test78-jantomay2022-16tb7p-filter-v6-dd.min-mar2023.unmin.binpack \ filt-v6-dd/test78-juntosep2022-16tb7p-filter-v6-dd.binpack \ filt-v6-dd/test77-dec2021-16tb7p-filter-v6-dd.binpack \ test80-apr2023-2tb7p.binpack \ /data/leela96-dfrc99-T60novdec-v2-T80juntonovjanfebT79aprmayT78jantosepT77dec-v6dd-T80apr.binpack ``` T80 apr2023 data was converted using lc0-rescorer with ~2tb of tablebases and can be found at: https://robotmoon.com/nnue-training-data/ Local elo at 25k nodes per move vs. nn-e1fb1ade4432.nnue (L1 size 1024): nn-epoch559.nnue : 25.7 +/- 1.6 Passed STC: https://tests.stockfishchess.org/tests/view/647cd3b87cf638f0f53f9cbb LLR: 2.95 (-2.94,2.94) <0.00,2.00> Total: 59200 W: 16000 L: 15660 D: 27540 Ptnml(0-2): 159, 6488, 15996, 6768, 189 Passed LTC: https://tests.stockfishchess.org/tests/view/647d58de726f6b400e4085d8 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 58800 W: 16002 L: 15657 D: 27141 Ptnml(0-2): 44, 5607, 17748, 5962, 39 closes https://github.com/official-stockfish/Stockfish/pull/4606 bench 2141197