DEXSeq and parallel, orphan processes.
1
0
Entering edit mode
Bin Ye ▴ 30
@bin-ye-4715
Last seen 10.3 years ago
Dear list, I've been using DEXSeq as describe in vignette on our SGE cluster which worked as expected, but I noticed that when any command with nCore=N (N > 1) called, there will be N orphan processes left in background. After the command finished, those N processes will be in sleep and hold certain amount of memory, but 0 cpu. These just keep adding up until it crashes the compute node. I've tried to clean-up with multicore command kill(children()) in R, it just return NULL without doing anything. I had to manually kill in linux command line. And I can't find in vignette how to deal with this. Is this an issue with package parallel/multicore? My session info is attached below. Is there a way to clean-up in R after calling those multi-processes commands? It will be really helpful if the vignette can be updated on this issue. Thank you! Bin > sessionInfo() R version 2.15.1 (2012-06-22) Platform: x86_64-redhat-linux-gnu (64-bit) locale: [1] LC_CTYPE=en_US.iso885915 LC_NUMERIC=C [3] LC_TIME=en_US.iso885915 LC_COLLATE=en_US.iso885915 [5] LC_MONETARY=en_US.iso885915 LC_MESSAGES=en_US.iso885915 [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.iso885915 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] DEXSeq_1.4.0 AnnotationDbi_1.20.3 Biobase_2.18.0 [4] BiocGenerics_0.4.0 RSQLite_0.11.2 DBI_0.2-5 loaded via a namespace (and not attached): [1] annotate_1.36.0 biomaRt_2.14.0 clusterProfiler_1.6.0 [4] colorspace_1.2-0 DESeq_1.10.1 dichromat_1.2-4 [7] digest_0.5.2 DO.db_2.5.0 DOSE_1.4.0 [10] genefilter_1.40.0 geneplotter_1.36.0 ggplot2_0.9.3 [13] GO.db_2.8.0 GOSemSim_1.16.1 grid_2.15.1 [16] gtable_0.1.2 hwriter_1.3 igraph_0.5.5-4 [19] IRanges_1.16.4 KEGG.db_2.8.0 labeling_0.1 [22] lattice_0.20-10 MASS_7.3-22 munsell_0.4 [25] parallel_2.15.1 plyr_1.8 proto_0.3-9.2 [28] qvalue_1.32.0 RColorBrewer_1.0-5 RCurl_1.95-3 [31] reshape2_1.2.2 scales_0.2.3 splines_2.15.1 [34] statmod_1.4.16 stats4_2.15.1 stringr_0.6.2 [37] survival_2.36-14 tcltk_2.15.1 tools_2.15.1 [40] XML_3.95-0.1 xtable_1.7-0 [[alternative HTML version deleted]]
GO DEXSeq GO DEXSeq • 1.1k views
ADD COMMENT
0
Entering edit mode
Alejandro Reyes ★ 1.9k
@alejandro-reyes-5124
Last seen 5 months ago
Novartis Institutes for BioMedical Reseā€¦
Dear Bin Ye, I do not know exactly where the problem is coming from, but my guess would be that it is a problem is from the parallel package. Does it happen only when you use DEXSeq? Or it also happens when you use parallel in general? Best regards, Alejandro Reyes > Dear list, > > I've been using DEXSeq as describe in vignette on our SGE cluster which > worked as expected, but I noticed that when any command with nCore=N (N > 1) > called, there will be N orphan processes left in background. After the > command finished, those N processes will be in sleep and hold certain amount > of memory, but 0 cpu. These just keep adding up until it crashes the compute > node. I've tried to clean-up with multicore command kill(children()) in R, > it just return NULL without doing anything. I had to manually kill in linux > command line. And I can't find in vignette how to deal with this. Is this an > issue with package parallel/multicore? My session info is attached below. > > Is there a way to clean-up in R after calling those multi-processes > commands? It will be really helpful if the vignette can be updated on this > issue. > > Thank you! > > Bin > >> sessionInfo() > R version 2.15.1 (2012-06-22) > Platform: x86_64-redhat-linux-gnu (64-bit) > > locale: > [1] LC_CTYPE=en_US.iso885915 LC_NUMERIC=C > [3] LC_TIME=en_US.iso885915 LC_COLLATE=en_US.iso885915 > [5] LC_MONETARY=en_US.iso885915 LC_MESSAGES=en_US.iso885915 > [7] LC_PAPER=C LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.iso885915 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] DEXSeq_1.4.0 AnnotationDbi_1.20.3 Biobase_2.18.0 > [4] BiocGenerics_0.4.0 RSQLite_0.11.2 DBI_0.2-5 > > loaded via a namespace (and not attached): > [1] annotate_1.36.0 biomaRt_2.14.0 clusterProfiler_1.6.0 > [4] colorspace_1.2-0 DESeq_1.10.1 dichromat_1.2-4 > [7] digest_0.5.2 DO.db_2.5.0 DOSE_1.4.0 > [10] genefilter_1.40.0 geneplotter_1.36.0 ggplot2_0.9.3 > [13] GO.db_2.8.0 GOSemSim_1.16.1 grid_2.15.1 > [16] gtable_0.1.2 hwriter_1.3 igraph_0.5.5-4 > [19] IRanges_1.16.4 KEGG.db_2.8.0 labeling_0.1 > [22] lattice_0.20-10 MASS_7.3-22 munsell_0.4 > [25] parallel_2.15.1 plyr_1.8 proto_0.3-9.2 > [28] qvalue_1.32.0 RColorBrewer_1.0-5 RCurl_1.95-3 > [31] reshape2_1.2.2 scales_0.2.3 splines_2.15.1 > [34] statmod_1.4.16 stats4_2.15.1 stringr_0.6.2 > [37] survival_2.36-14 tcltk_2.15.1 tools_2.15.1 > [40] XML_3.95-0.1 xtable_1.7-0 > > > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
ADD COMMENT
0
Entering edit mode
Hi Alejandro, Yes, I think it's something with parallel. There's other user met similar problem with package zinba, which also based on parallel. I'll try r-help list, see if anyone there can help. Thank you! Bin On 1/29/13 12:57 PM, "Alejandro Reyes" <alejandro.reyes at="" embl.de=""> wrote: >Dear Bin Ye, > >I do not know exactly where the problem is coming from, but my guess >would be that it is a problem is from the parallel package. Does it >happen only when you use DEXSeq? Or it also happens when you use >parallel in general? > >Best regards, >Alejandro Reyes > >> Dear list, >> >> I've been using DEXSeq as describe in vignette on our SGE cluster which >> worked as expected, but I noticed that when any command with nCore=N (N >>> 1) >> called, there will be N orphan processes left in background. After the >> command finished, those N processes will be in sleep and hold certain >>amount >> of memory, but 0 cpu. These just keep adding up until it crashes the >>compute >> node. I've tried to clean-up with multicore command kill(children()) in >>R, >> it just return NULL without doing anything. I had to manually kill in >>linux >> command line. And I can't find in vignette how to deal with this. Is >>this an >> issue with package parallel/multicore? My session info is attached >>below. >> >> Is there a way to clean-up in R after calling those multi-processes >> commands? It will be really helpful if the vignette can be updated on >>this >> issue. >> >> Thank you! >> >> Bin >> >>> sessionInfo() >> R version 2.15.1 (2012-06-22) >> Platform: x86_64-redhat-linux-gnu (64-bit) >> >> locale: >> [1] LC_CTYPE=en_US.iso885915 LC_NUMERIC=C >> [3] LC_TIME=en_US.iso885915 LC_COLLATE=en_US.iso885915 >> [5] LC_MONETARY=en_US.iso885915 LC_MESSAGES=en_US.iso885915 >> [7] LC_PAPER=C LC_NAME=C >> [9] LC_ADDRESS=C LC_TELEPHONE=C >> [11] LC_MEASUREMENT=en_US.iso885915 LC_IDENTIFICATION=C >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> other attached packages: >> [1] DEXSeq_1.4.0 AnnotationDbi_1.20.3 Biobase_2.18.0 >> [4] BiocGenerics_0.4.0 RSQLite_0.11.2 DBI_0.2-5 >> >> loaded via a namespace (and not attached): >> [1] annotate_1.36.0 biomaRt_2.14.0 clusterProfiler_1.6.0 >> [4] colorspace_1.2-0 DESeq_1.10.1 dichromat_1.2-4 >> [7] digest_0.5.2 DO.db_2.5.0 DOSE_1.4.0 >> [10] genefilter_1.40.0 geneplotter_1.36.0 ggplot2_0.9.3 >> [13] GO.db_2.8.0 GOSemSim_1.16.1 grid_2.15.1 >> [16] gtable_0.1.2 hwriter_1.3 igraph_0.5.5-4 >> [19] IRanges_1.16.4 KEGG.db_2.8.0 labeling_0.1 >> [22] lattice_0.20-10 MASS_7.3-22 munsell_0.4 >> [25] parallel_2.15.1 plyr_1.8 proto_0.3-9.2 >> [28] qvalue_1.32.0 RColorBrewer_1.0-5 RCurl_1.95-3 >> [31] reshape2_1.2.2 scales_0.2.3 splines_2.15.1 >> [34] statmod_1.4.16 stats4_2.15.1 stringr_0.6.2 >> [37] survival_2.36-14 tcltk_2.15.1 tools_2.15.1 >> [40] XML_3.95-0.1 xtable_1.7-0 >> >> >> >> >> [[alternative HTML version deleted]] >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: >>http://news.gmane.org/gmane.science.biology.informatics.conductor >
ADD REPLY

Login before adding your answer.

Traffic: 434 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6