I'm trying to run bumphunter within the minfi package on a linux cluster, and when I use doParallel with bumphunter it consistently errors out with "Error in mcfork() : unable to fork, possible reason: Cannot allocate memory" I haven't been able to find a solution searching the web or working with our support staff. Has anyone seen this before or know where the problem is? Any help will be greatly appreciated. - Brad -- output of sessionInfo(): Loading module hpc/intel-mkl-11.0.0.079. JDK 1.6 module **************************************************** This module sets up the following environment variables for JDK 1.6: PATH /n/app/jdk1.6.0_11/bin **************************************************** Loading module centos6/tcl-8.5.14. Loading module centos6/tk-8.5.14. Loading module centos6/fftw-3.3_gcc-4.4.7. Loading module centos6/gsl-1.16_gcc-4.4.7. Loading module centos6/hdf5-1.8.11_gcc-4.4.7. Loading module centos6/netcdf-4.3.0_gcc-4.4.7. Loading module centos6/R-3.0.2. Loading required package: minfi Loading required package: methods Loading required package: BiocGenerics Loading required package: parallel Attaching package: ???????BiocGenerics???????? The following objects are masked from ???????package:parallel????????: clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, clusterExport, clusterMap, parApply, parCapply, parLapply, parLapplyLB, parRapply, parSapply, parSapplyLB The following object is masked from ???????package:stats????????: xtabs The following objects are masked from ???????package:base????????: anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, duplicated, eval, evalq, Filter, Find, get, intersect, is.unsorted, lapply, Map, mapply, match, mget, order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank, rbind, Reduce, rep.int, rownames, sapply, setdiff, sort, table, tapply, union, unique, unlist Loading required package: Biobase Welcome to Bioconductor Vignettes contain introductory material; view with 'browseVignettes()'. To cite Bioconductor, see 'citation("Biobase")', and for packages 'citation("pkgname")'. Loading required package: lattice Loading required package: reshape Loading required package: plyr Attaching package: ???????reshape???????? The following objects are masked from ???????package:plyr????????: rename, round_any Loading required package: GenomicRanges Loading required package: IRanges Attaching package: ???????IRanges???????? The following objects are masked from ???????package:reshape????????: expand, rename The following objects are masked from ???????package:plyr????????: desc, rename Loading required package: XVector Attaching package: ???????XVector???????? The following object is masked from ???????package:plyr????????: compact Loading required package: Biostrings Loading required package: bumphunter Loading required package: foreach Loading required package: iterators Loading required package: locfit locfit 1.5-9.1 2013-03-22 Attaching package: ???????locfit???????? The following objects are masked from ???????package:GenomicRanges????????: left, right Loading required package: IlluminaHumanMethylation450kmanifest Loading required package: IlluminaHumanMethylation450kanno.ilmn12.hg19 [bumphunterEngine] Parallelizing using 64 workers/cores (backend: doParallel, version: 1.0.6). [bumphunterEngine] Computing coefficients. [bumphunterEngine] Performing 1000 permutations. [bumphunterEngine] Computing marginal permutation p-values. [bumphunterEngine] cutoff: 0.05 [bumphunterEngine] Finding regions. [bumphunterEngine] Found 175279 bumps. [bumphunterEngine] Computing regions for each permutation. Loading required package: rngtools Loading required package: pkgmaker Loading required package: registry Attaching package: ???????pkgmaker???????? The following object is masked from ???????package:IRanges????????: new2 Error in mcfork() : unable to fork, possible reason: Cannot allocate memory Calls: bumphunter ... %dopar% -> <anonymous> -> mclapply -> lapply -> FUN -> mcfork Execution halted -- Sent via the guest posting facility at bioconductor.org.

On Sat, Apr 12, 2014 at 10:42 PM, Brad Ruzicka [guest] < guest@bioconductor.org> wrote: > > I'm trying to run bumphunter within the minfi package on a linux cluster, > and when I use doParallel with bumphunter it consistently errors out with > "Error in mcfork() : unable to fork, possible reason: Cannot allocate > memory" > I haven't been able to find a solution searching the web or working with > our support staff. > Has anyone seen this before or know where the problem is? > Any help will be greatly appreciated. > - Brad > > > -- output of sessionInfo(): > > Loading module hpc/intel-mkl-11.0.0.079. > > JDK 1.6 module > **************************************************** > > This module sets up the following environment > variables for JDK 1.6: > PATH /n/app/jdk1.6.0_11/bin > > **************************************************** > > Loading module centos6/tcl-8.5.14. > Loading module centos6/tk-8.5.14. > Loading module centos6/fftw-3.3_gcc-4.4.7. > Loading module centos6/gsl-1.16_gcc-4.4.7. > Loading module centos6/hdf5-1.8.11_gcc-4.4.7. > Loading module centos6/netcdf-4.3.0_gcc-4.4.7. > Loading module centos6/R-3.0.2. > Loading required package: minfi > Loading required package: methods > Loading required package: BiocGenerics > Loading required package: parallel > > Attaching package: ‘BiocGenerics’ > > The following objects are masked from ‘package:parallel’: > > clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, > clusterExport, clusterMap, parApply, parCapply, parLapply, > parLapplyLB, parRapply, parSapply, parSapplyLB > > The following object is masked from ‘package:stats’: > > xtabs > > The following objects are masked from ‘package:base’: > > anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, > duplicated, eval, evalq, Filter, Find, get, intersect, is.unsorted, > lapply, Map, mapply, match, mget, order, paste, pmax, pmax.int, > pmin, pmin.int, Position, rank, rbind, Reduce, rep.int, rownames, > sapply, setdiff, sort, table, tapply, union, unique, unlist > > Loading required package: Biobase > Welcome to Bioconductor > > Vignettes contain introductory material; view with > 'browseVignettes()'. To cite Bioconductor, see > 'citation("Biobase")', and for packages 'citation("pkgname")'. > > Loading required package: lattice > Loading required package: reshape > Loading required package: plyr > > Attaching package: ‘reshape’ > > The following objects are masked from ‘package:plyr’: > > rename, round_any > > Loading required package: GenomicRanges > Loading required package: IRanges > > Attaching package: ‘IRanges’ > > The following objects are masked from ‘package:reshape’: > > expand, rename > > The following objects are masked from ‘package:plyr’: > > desc, rename > > Loading required package: XVector > > Attaching package: ‘XVector’ > > The following object is masked from ‘package:plyr’: > > compact > > Loading required package: Biostrings > Loading required package: bumphunter > Loading required package: foreach > Loading required package: iterators > Loading required package: locfit > locfit 1.5-9.1 2013-03-22 > > Attaching package: ‘locfit’ > > The following objects are masked from ‘package:GenomicRanges’: > > left, right > > Loading required package: IlluminaHumanMethylation450kmanifest > Loading required package: IlluminaHumanMethylation450kanno.ilmn12.hg19 > [bumphunterEngine] Parallelizing using 64 workers/cores (backend: > doParallel, version: 1.0.6). > i don't think it is possible to say exactly how to get around this on the basis of this information. You have to match the request to the available resources. One theory: at some point you ran registerDoParallel(cores=64) to get the message above. The image that is forked has memory footprint F, say, and the machine that will run the job must have at least 64*F memory available to you to succeed with this request. If you reduce the request on the number of cores, you may complete the job; I don't know enough about the whole setup to be more precise. Another theory -- you have a cluster of 64 machines but at least one of them does not have enough RAM to satisfy whatever mcfork requests occured there. In principle you can shape the request to use multiple cores on multiple independent machines but you have to respect the limits of the various resources. Experimenting with a partial run to estimate what will be needed is a useful and often essential step. > [bumphunterEngine] Computing coefficients. > [bumphunterEngine] Performing 1000 permutations. > [bumphunterEngine] Computing marginal permutation p-values. > [bumphunterEngine] cutoff: 0.05 > [bumphunterEngine] Finding regions. > [bumphunterEngine] Found 175279 bumps. > [bumphunterEngine] Computing regions for each permutation. > Loading required package: rngtools > Loading required package: pkgmaker > Loading required package: registry > > Attaching package: ‘pkgmaker’ > > The following object is masked from ‘package:IRanges’: > > new2 > > Error in mcfork() : > unable to fork, possible reason: Cannot allocate memory > Calls: bumphunter ... %dopar% -> <anonymous> -> mclapply -> lapply -> FUN > -> mcfork > Execution halted > > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > [[alternative HTML version deleted]]

Send your sessionInfo(). While you're doing that, here are some suggestions: 1) use fewer processes. minfi will kill a machine with 64GB of RAM using as few as 4 processes for bumphunting. Plus, you probably want to use doMC() instead of doParallel(). Try setting options("mc.cores") to something like 4 and gradually increasing it as you successfully run analyses on this machine/cluster/whatever. 2) look into using DMRcate if it's a simple 1:1 comparison. It's faster than bumphunter. 3) the real drag here is that it looks like your run actually completes!! You have a huge number of candidate bumps but when the permutation testing is being run, you run out of memory for at least one of the child processes, and that's all she wrote. So, back to #1, use fewer processes for the time being. If you can put more RAM into the machine, by all means do so, but until that is done, try using a relatively small number of processes. Best, --t Statistics is the grammar of science. Karl Pearson <http: en.wikipedia.org="" wiki="" the_grammar_of_science=""> On Sat, Apr 12, 2014 at 7:42 PM, Brad Ruzicka [guest] < guest@bioconductor.org> wrote: > > I'm trying to run bumphunter within the minfi package on a linux cluster, > and when I use doParallel with bumphunter it consistently errors out with > "Error in mcfork() : unable to fork, possible reason: Cannot allocate > memory" > I haven't been able to find a solution searching the web or working with > our support staff. > Has anyone seen this before or know where the problem is? > Any help will be greatly appreciated. > - Brad > > > -- output of sessionInfo(): > > Loading module hpc/intel-mkl-11.0.0.079. > > JDK 1.6 module > **************************************************** > > This module sets up the following environment > variables for JDK 1.6: > PATH /n/app/jdk1.6.0_11/bin > > **************************************************** > > Loading module centos6/tcl-8.5.14. > Loading module centos6/tk-8.5.14. > Loading module centos6/fftw-3.3_gcc-4.4.7. > Loading module centos6/gsl-1.16_gcc-4.4.7. > Loading module centos6/hdf5-1.8.11_gcc-4.4.7. > Loading module centos6/netcdf-4.3.0_gcc-4.4.7. > Loading module centos6/R-3.0.2. > Loading required package: minfi > Loading required package: methods > Loading required package: BiocGenerics > Loading required package: parallel > > Attaching package: ââ'¬ËoeBiocGenericsââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:parallelââ'¬â"¢: > > clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, > clusterExport, clusterMap, parApply, parCapply, parLapply, > parLapplyLB, parRapply, parSapply, parSapplyLB > > The following object is masked from ââ'¬Ëoepackage:statsââ'¬â"¢: > > xtabs > > The following objects are masked from ââ'¬Ëoepackage:baseââ'¬â"¢: > > anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, > duplicated, eval, evalq, Filter, Find, get, intersect, is.unsorted, > lapply, Map, mapply, match, mget, order, paste, pmax, pmax.int, > pmin, pmin.int, Position, rank, rbind, Reduce, rep.int, rownames, > sapply, setdiff, sort, table, tapply, union, unique, unlist > > Loading required package: Biobase > Welcome to Bioconductor > > Vignettes contain introductory material; view with > 'browseVignettes()'. To cite Bioconductor, see > 'citation("Biobase")', and for packages 'citation("pkgname")'. > > Loading required package: lattice > Loading required package: reshape > Loading required package: plyr > > Attaching package: ââ'¬Ëoereshapeââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:plyrââ'¬â"¢: > > rename, round_any > > Loading required package: GenomicRanges > Loading required package: IRanges > > Attaching package: ââ'¬ËoeIRangesââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:reshapeââ'¬â"¢: > > expand, rename > > The following objects are masked from ââ'¬Ëoepackage:plyrââ'¬â"¢: > > desc, rename > > Loading required package: XVector > > Attaching package: ââ'¬ËoeXVectorââ'¬â"¢ > > The following object is masked from ââ'¬Ëoepackage:plyrââ'¬â"¢: > > compact > > Loading required package: Biostrings > Loading required package: bumphunter > Loading required package: foreach > Loading required package: iterators > Loading required package: locfit > locfit 1.5-9.1 2013-03-22 > > Attaching package: ââ'¬Ëoelocfitââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:GenomicRangesââ'¬â"¢: > > left, right > > Loading required package: IlluminaHumanMethylation450kmanifest > Loading required package: IlluminaHumanMethylation450kanno.ilmn12.hg19 > [bumphunterEngine] Parallelizing using 64 workers/cores (backend: > doParallel, version: 1.0.6). > [bumphunterEngine] Computing coefficients. > [bumphunterEngine] Performing 1000 permutations. > [bumphunterEngine] Computing marginal permutation p-values. > [bumphunterEngine] cutoff: 0.05 > [bumphunterEngine] Finding regions. > [bumphunterEngine] Found 175279 bumps. > [bumphunterEngine] Computing regions for each permutation. > Loading required package: rngtools > Loading required package: pkgmaker > Loading required package: registry > > Attaching package: ââ'¬Ëoepkgmakerââ'¬â"¢ > > The following object is masked from ââ'¬Ëoepackage:IRangesââ'¬â"¢: > > new2 > > Error in mcfork() : > unable to fork, possible reason: Cannot allocate memory > Calls: bumphunter ... %dopar% -> <anonymous> -> mclapply -> lapply -> FUN > -> mcfork > Execution halted > > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > [[alternative HTML version deleted]]