My question is actually simple: How can I manage to create several DoparParam()
objects, each with a different number of registered cores? See below for what I tried. In a nutshell, it seems I can only have one DoparParam()
object, because once I run registerDoParallel()
existing objects will be overwritten in terms of number of registred cores.
Why is that important? Say I need one DoparParam with just very few cores, because the bplapply will be very memory-intensive, and one with many cores, because the bplapply will not be memory-intensive at all. How can I do that? Please do not suggest to use any of the other available backends. DoparParam (in my hands) works the smoothest, on all OS, with least problems, so I really would like to use that, but in a more flexible fashion in terms of having multiple ones available with different numbers of cores, and without running registerDoParallel
before each bplapply
, but rather once in a setup script, and then go along with the objects all the way through the project.
suppressMessages({
library(BiocParallel)
library(doParallel)
})
registerDoParallel(cores = 2)
bp2 <- DoparParam()
bp2
#> class: DoparParam
#> bpisup: TRUE; bpnworkers: 2; bptasks: 0; bpjobname: BPJOB
#> bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#> bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#> bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#> bpfallback: TRUE
#> bplogdir: NA
#> bpresultdir: NA
# no effect, still using 2 cores as workers
bp2$workers <- 18L
bp2
#> class: DoparParam
#> bpisup: TRUE; bpnworkers: 2; bptasks: 0; bpjobname: BPJOB
#> bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#> bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#> bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#> bpfallback: TRUE
#> bplogdir: NA
#> bpresultdir: NA
# no effect, still using 2 cores as workers
bptasks(bp2) <- 18L
bp2
#> class: DoparParam
#> bpisup: TRUE; bpnworkers: 2; bptasks: 18; bpjobname: BPJOB
#> bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#> bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#> bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#> bpfallback: TRUE
#> bplogdir: NA
#> bpresultdir: NA
# only valid for some Snow and MultiCore
bpworkers(bp2) <- 18L
#> Error in (function (classes, fdef, mtable) : unable to find an inherited method for function 'bpworkers<-' for signature '"DoparParam", "integer"'
# overwrites bp2
registerDoParallel(cores = 18)
bp18 <- DoparParam()
bp18
#> class: DoparParam
#> bpisup: TRUE; bpnworkers: 18; bptasks: 0; bpjobname: BPJOB
#> bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#> bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#> bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#> bpfallback: TRUE
#> bplogdir: NA
#> bpresultdir: NA
bp2
#> class: DoparParam
#> bpisup: TRUE; bpnworkers: 18; bptasks: 18; bpjobname: BPJOB
#> bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#> bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#> bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#> bpfallback: TRUE
#> bplogdir: NA
#> bpresultdir: NA
sessionInfo()
#> R version 4.3.2 (2023-10-31)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 22.04.3 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] parallel stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] doParallel_1.0.17 iterators_1.0.14 foreach_1.5.2
#> [4] BiocParallel_1.36.0
#>
#> loaded via a namespace (and not attached):
#> [1] digest_0.6.33 codetools_0.2-19 fastmap_1.1.1 xfun_0.41
#> [5] glue_1.6.2 knitr_1.45 htmltools_0.5.7 rmarkdown_2.25
#> [9] lifecycle_1.0.4 cli_3.6.1 reprex_2.0.2 withr_2.5.2
#> [13] compiler_4.3.2 rstudioapi_0.15.0 tools_4.3.2 evaluate_0.23
#> [17] yaml_2.3.7 rlang_1.1.2 fs_1.6.3
Created on 2024-06-17 with reprex v2.0.2
This does not answer my question. I am not aiming to register different types of backends, but the same backend with different configurations. BiocParallel is obviously a Bioc package, so it's on topic.
registerDoParallel
andDoparParam
are bothdoParallel
functions, which again, is a CRAN package. What part of the code you present above is meant to be used withinBiocParallel
?https://github.com/Bioconductor/BiocParallel/blob/devel/R/DoparParam-class.R
DoparParam is a BiocParallel class, that is why I am asking here.
OK, my bad. Never used that aspect of the package before.