I am running VariantTools on 40 not particularly large bam files (~700 Mb). When running 5 in parallel on the same machine (12 cores, 48 Gig RAM), each running on a single core, it took several hours for the first one to finish, and the other 4 are still running. In total, the remaing 4 are taking up 70% of the RAM. Does anyone have suggestions for how to run VariantTools for optimum speed? I can increase the number of cores, but then I have to run fewer files because otherwise I run out of memory. And if I run 5 at a time each with a single core, I'm coming close to using up all the memory, and I don't know if that will really slow things down and if it would therefore be better to run just 4 at a time. Any suggestions are appreciated. Thank you. Eric -- output of sessionInfo(): These are running in batch mode so I don't have that information. -- Sent via the guest posting facility at bioconductor.org.

I think you should try increasing the cores (for the tally step). The memory consumption in that step is mostly due to the summaries. All the summaries need to be generated prior to the filtering, so you might as well calculate them more quickly. Another idea is to parallelize at a higher level, so that each core is responsible for both tallying and filtering (for, say, one chromosome). This would be more work on your part, though. Eventually VariantTools needs to make this easier. Michael On Thu, Jun 13, 2013 at 2:02 PM, Eric Foss [guest] <guest@bioconductor.org>wrote: > > I am running VariantTools on 40 not particularly large bam files (~700 > Mb). When running 5 in parallel on the same machine (12 cores, 48 Gig RAM), > each running on a single core, it took several hours for the first one to > finish, and the other 4 are still running. In total, the remaing 4 are > taking up 70% of the RAM. Does anyone have suggestions for how to run > VariantTools for optimum speed? I can increase the number of cores, but > then I have to run fewer files because otherwise I run out of memory. And > if I run 5 at a time each with a single core, I'm coming close to using up > all the memory, and I don't know if that will really slow things down and > if it would therefore be better to run just 4 at a time. Any suggestions > are appreciated. > > Thank you. > > Eric > > -- output of sessionInfo(): > > These are running in batch mode so I don't have that information. > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > [[alternative HTML version deleted]]