Improved Linux shared memory implementation
Our existing Linux shm implementation has worked quite well since its introduction after SF 17.1, for example substantially improving fishtest throughput. It does have a few pain points:
- /dev/shm needs to have enough space for the network, some 100 MB;
- some of the filesystem operations are racy, so sharing can non-deterministically fail;
- the network is never cleaned up on SIGKILL. Obviously that's the fault of the user, but OpenBench uses SIGKILL, because some engines misbehave.
This prototype is a fleshed-out version of Sopel's comment https://github.com/official-stockfish/Stockfish/pull/6979#issuecomment-4986789088
## The implementation
As usual we derive the shared memory name from the network contents, executable path and the NUMA node. Fishes rendezvous at a shared folder `/tmp/stockfish-[uid]/sfshm_...`, which has the following structure:
```
/tmp/stockfish-[uid]/sfshm_...
├─ init_lock
├─ [pid1].sock
│ ...
└─ [pid2].sock
```
An anonymous file containing the network is first created, sized, and initialized via `memfd_create` by a unique instance of Stockfish. This creator instance then sends the file descriptor using Unix sockets to other Stockfish instances. Each of those instances is also capable of sending the memfd to any other instance, so if the original creator instance exits, new instances can still receive access.
More specifically, all *initialized* shared memory instances have a background thread (see `make_server_thread`) which listens to the socket at `/tmp/stockfish-[uid]/sfshm_.../[pid].sock`. When a requestor sends a message to this socket, the background thread replies with the network memfd. The requestor can then map the fd, sharing the underlying memory with the other instances.
### Initialization (`SharedMemory::open`)
The implementation first `flock`s the `init_lock` file to get an `InitLock`. While holding the lock, it:
- enumerates all sockets in the shared folder;
- attempts to receive the memfd from each of those sockets, stopping on the first success. (We also delete sockets that are found to belong to dead processes.)
If a memfd is received, it is mapped and the lock is released. If no memfd is found, then this process is the creator. We continue to hold the lock, call `memfd_create` and initialize it with the network (decompressed from disk). We then open the socket (guaranteed to have a unique name using our PID), and spawn the background server thread that listens on that socket. Finally, we release the lock.
### De-initialization
When a `SharedMemory<T>` is closed, the background thread is joined, the memory is unmapped, and file descriptors are closed. We also unlink the `.sock` file. When *exiting* the program, we skip joining the background thread and unmapping memory because these actions are unnecessary and could cause problems if called in an exit or signal handler.
## Bookkeeping
Passed non-regression STC
LLR: 3.04 (-2.94,2.94) <-1.75,0.25>
Total: 252192 W: 65179 L: 65187 D: 121826
Ptnml(0-2): 549, 26214, 72561, 26240, 532
https://tests.stockfishchess.org/tests/view/6a63b2dac054e285ae0287f3
Fixes #6977
closes https://github.com/official-stockfish/Stockfish/pull/6988
No functional change