Hi,

I have a question which is especially addressed to the authors of the "BiocParallel" package: In the vignette "Introduction to BiocParallel", you state (at the top of page 13):

Create a SnowParam instance with the number of nodes equal to the size of the MPI universe minus 1 (let one node dispatch jobs to workers) [...]

The corresponding code starts with:

param <- SnowParam(mpi.universe.size() - 1, "MPI")

My question is: Do you have a reference for the rule that one has to take minus 1 instead of just the whole MPI universe? When I just use the whole MPI universe, I don't run into any problems.

Thanks and best regards!

Usually MPI programs are though of as 'data parallel', where the work W is divided between n nodes. In contrast the BiocParallel paradigm is of 'manager / worker', where one nodes plays the role of manager -- starting the workers, sending tasks to them, collecting and collating results. If there are n nodes available, and 1 node must be used by the manager, there are n - 1 nodes left to act as worker. Using n nodes as worker means that the processor associated with the manager is doing double-duty, managing all the other workers and doing it's own work. This 'over-subscription' could have consequences for performance, but it is not surprising that, in the context of R, no meaningful difference is observed.