Hello,

I am replacing foreach with BiocParallel in my package. I wonder whether could I maintain the same setting of progress bar  as in foreach for bplapply. (The same problem as listed in https://github.com/Bioconductor/BiocParallel/issues/54 and https://stat.ethz.ch/pipermail/bioc-devel/2017-December/012572.html.

Firstly I created a simple example in R:

nrow=10000
ncol=500
matrixx=matrix(runif(nrow*ncol),nrow=nrow,ncol=ncol)

Using foreach with progressbar:

library(parallel)
library(doSNOW)
library(foreach)
cluster=makeCluster(5,type='SOCK')
registerDoSNOW(cluster)
getDoParWorkers()
iterations<-nrow
pb<-txtProgressBar(max=iterations,style =3)
progress<-function(n)setTxtProgressBar(pb,n)
opts<-list(progress=progress)
BB_parmat<-foreach(geneind=1:dim(matrixx)[1],.combine=c,.options.snow=opts)%dopar%{
  return(mean(matrixx[geneind,]))
}

close(pb)
stopCluster(cluster)

Using bplapply with progress bar (a potential problem is that the progressbar will show 0% for a long time, and then suddenly increases):

library(BiocParallel)
BPPARAM=SnowParam(workers=5,progressbar = TRUE,type='SOCK')
funnn<-function(geneind,matrixx){
  return(mean(matrixx[geneind,]))
}

suppressWarnings(temp_result<-bplapply(seq(1,dim(matrixx)[1]),funnn,matrixx,BPPARAM=BPPARAM))

I prefer the progress bar shown in the foreach  case: increase the bar by 1% per time, so that I can have a basic idea about the running time of the whole code. In the second case, the progress bar increases suddenly. 

My question is how could I achieve the same progress bar as shown in foreach case using bplapply? 

Thank you very much!

Best wishes,

Wenhao

The effect can be achieved by setting the number of tasks, e.g.,

BPPARAM=SnowParam(workers=5, tasks = 20, progressbar = TRUE,type='SOCK')

updates the progress bar 20 times.

The way bplapply works is that, by default, it splits the initial task list (in your case the sequence of row indexes) into equal components for each worker -- each worker gets 10000 / 5 = 2000 rows. These are sent to the workers, who report back when done. When each worker finishes, the progress bar advances. The progress bar advances in 5 steps, but since the workers all finish at about the same time it seems like the progress bar jumps to complete.

The effect of setting tasks = 20 is to divide the 10000 tasks into 10000 / 20 = 500 rows per task, to send 500 x 5 to the first five workers, and as each worker finishes the progress bar is updated and the next 500 tasks sent to the worker. The progress bar moves across the screen more smoothly, but actually the computation is less efficient (because there is more communication between the manager and workers) and takes longer. If most of the time is spent in computation anyway, then the extra cost of communication is small and the trade-off may be worth it.

Usually of course it is better to vectorize than to parallelize, so in the above trivial example simply rowMeans(matrixx).

(the comment on your question was from a spammer, and was deleted).