Neural Network 1536 vs 2560

[Equalizer-0]

I have compared different network sizes and this seems to be the best performing.
Is the 2560 network really necessary or is it more useful to continue the development of the 1536?

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 5269 - 3967 - 10764 [0.533] 20000 - tc=1.0+0.01
... Stockfish_240307_1536_nn-78bacfcee510 playing White: 3345 - 1812 - 4843 [0.577] 10000
... Stockfish_240307_1536_nn-78bacfcee510 playing Black: 1924 - 2155 - 5921 [0.488] 10000
... White vs Black: 5500 - 3736 - 10764 [0.544] 20000
Elo difference: 22.7 +/- 3.3, LOS: 100.0 %, DrawRatio: 53.8 %

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 106 - 90 - 1804 [0.504] 2000 - tc=10.0+0.1
... Stockfish_240307_1536_nn-78bacfcee510 playing White: 70 - 53 - 877 [0.508] 1000
... Stockfish_240307_1536_nn-78bacfcee510 playing Black: 36 - 37 - 927 [0.499] 1000
... White vs Black: 107 - 89 - 1804 [0.504] 2000
Elo difference: 2.8 +/- 4.8, LOS: 87.3 %, DrawRatio: 90.2 %

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 0 - 2 - 798 [0.499] 800 - tc=180.0+3.0
... Stockfish_240307_1536_nn-78bacfcee510 playing White: 0 - 1 - 399 [0.499] 400
... Stockfish_240307_1536_nn-78bacfcee510 playing Black: 0 - 1 - 399 [0.499] 400
... White vs Black: 1 - 1 - 798 [0.500] 800
Elo difference: -0.9 +/- 1.2, LOS: 7.9 %, DrawRatio: 99.8 %

[mstembera]

If you really find the 1536 best performing please submit a test to Fishtest to show this. Note we care a lot more about LTC than STC.

[Equalizer-0]

After changing the CPU from 3700x 8/16 to 5950x 16/32, I realized that the tests I was running 100% using all threads were wrong.

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 5269 - 3967 - 10764 [0.533] 20000 - tc=1.0+0.01 - 3700x -concurrency 16 - cpu 100%

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 4113 - 4142 - 11745 [0.499] 20000 - tc=1.0+0.01 - 5950x -concurrency 16 - cpu 50%
tc=1.0+0.01 - 5950x -concurrency 32 - ERROR cutechess-cli does not test

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 3530 - 2289 - 4181 [0.562] 10000 - tc=1.0+0.01 - 5950x -concurrency 16 - 2 parallel cpu 50% + 50%
Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 3472 - 2376 - 4152 [0.555] 10000 - tc=1.0+0.01 - 5950x -concurrency 16 - 2 parallel cpu 50% + 50%

Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 2818 - 2247 - 4935 [0.529] 10000 - tc=2.0+0.02 - 5950x -concurrency 32 - cutechess-cli OK cpu 100%
Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 1353 - 1439 - 7208 [0.496] 10000 - tc=2.0+0.02 - 5950x -concurrency 16 - cpu 50%
Score of Stockfish_240307_1536_nn-78bacfcee510 vs Stockfish_240307_2560_nn-1ceb1ade0001: 1254 - 1308 - 7438 [0.497] 10000 - tc=2.0+0.02 - 5950x NO SMT -concurrency 16 - cpu 100%

I don't understand why if you use a concurrency greater than the number of cores the test will be incorrect.
Two parallel instances of -concurrency 16 report a very wrong result.

Why?