This is a post from 13 months ago; sorry it slipped through at the time. Here are current timings for me

9.275732 SerialParam
8.231515 MulticoreParam
2.275638 SnowParam, type = "FORK"
2.154827 mclapply

The default tasks already divides the work in the best way for this computation (and is the same strategy across each of the *Param).

bpvec() is not being used correctly. In a simplified example, if X = 1:10 were distributed over two workers, the first worker might receive 1:5 and the second worker 6:10. foo(1:5) is ok; length(foo(1:5)) == length(1:5). foo(6:10) is problematic because as written length(foo(6:10)) is 10 and not 5. An updated version is

foo <- function(idx) {
    res <- rep(0, length(idx))
    for(i in seq_along(idx))
        res[i] <- lm(Y[idx[i],]~X)$coef[2]
    res
}

This takes about 2.24 seconds when run with MulticoreParam() -- bpvec() would be even more useful if the implementation of foo() itself were 'vectorized', as e.g., in the trivial example in the help page ?bpvec where the body is sqrt(idx) rather than an iteration as here.

This catches us up to the question -- why is MulticoreParam being slow? In some respects it is implemented differently from mclapply() and so performance is not expected to be identical, but I'm surprised that SnowParam(type = "FORK") is relatively fast; I will look into this...

UPDATE The reason for the slowness is because MulticoreParam() includes a 'garbage collection' parameter force.GC set to TRUE by default. Setting this to FALSE recovers the performance of mclapply().

system.time({
    beta <- bplapply(1:N, foo, BPPARAM = MulticoreParam(4, force.GC = FALSE))
})
##   user  system elapsed 
##  9.330   0.180   2.455

There is a (pretty hard to follow grammatically, and not mentioning performance trade-offs) mention in the NEWS file

CHANGES IN VERSION 1.28
-----------------------

USER VISIBLE CHANGES
...
    o (v. 1.27.9) By defualt, do _not_ only run garbage collection
    after every call to FUN(), except under MulticoreParam(). R's
    garbage collection algorithm only fails to do well when forked
    processes (i.e., MulticoreParam) assume that they are the only
    consumers of process memory.

and in the commit log. As mentioned in the commit, the motivation was to manage memory better in response to this pull request. I will rethink the approach and have opened an issue to track this.

Returning to the original task of extracting coefficients from repeated fitting the same model matrix, note that it is much more efficient to use lm.fit() than to use lm(); I have

bpvec_lm <- function(idx) {
    ## better than `foo()` reimplemented above; does not require explicit
    ## memory management
    vapply(idx, function(i) {
        lm(Y[i,] ~ X)$coefficients[2]
    }, numeric(1))
}

bpvec_lmfit <- function(idx) {
    x <- cbind(1, X) # model matrix computed once instead of N times
    vapply(idx, function(i) {
        lm.fit(x, Y[i,])$coefficients[2]
    }, numeric(1))
}

system.time(res1 <- bpvec_lm(1:N))
##    user  system elapsed 
##   8.449   0.188   8.659 
system.time(res2 <- bpvec_lmfit(1:N))
##   user  system elapsed 
##   0.742   0.052   0.800 
system.time(res3 <- bpvec(1:N, bpvec_lmfit, BPPARAM = MulticoreParam(4)))
##    user  system elapsed 
##   0.826   0.093   0.274 
identical(res1, res2)
## TRUE
identical(res1, res3)
## TRUE

For a 10x speedup WITHOUT parallel evaluation.