Dear All,
I am facing some problems with MEDIPS package,
I Think I'am running out of memory once I run the command MEDIPS.meth
................
Adjusting p.values for multiple testing...
Creating results table...
Adding differential coverage results...
Error: cannot allocate vector of size 207.9 Mb
Inoltre: Warning messages:
1: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
Reached total allocation of 24501Mb: see help(memory.size)
2: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
Reached total allocation of 24501Mb: see help(memory.size)
3: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
Reached total allocation of 24501Mb: see help(memory.size)
4: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
Reached total allocation of 24501Mb: see help(memory.size)
Is there a way to avoid this problem?
I have a 24 Gb RAM workstation I don't see why it fails...
Thanks,
Paolo
> sessionInfo()
R version 3.0.0 (2013-04-03)
Platform: x86_64-w64-mingw32/x64 (64-bit)
locale:
[1] LC_COLLATE=Italian_Italy.1252 LC_CTYPE=Italian_Italy.1252
LC_MONETARY=Italian_Italy.1252 LC_NUMERIC=C
LC_TIME=Italian_Italy.1252
attached base packages:
[1] parallel stats graphics grDevices utils datasets
methods base
other attached packages:
[1] MEDIPS_1.10.0 BiocInstaller_1.10.3 gtools_3.1.0
DNAcopy_1.34.0 BSgenome_1.28.0 Biostrings_2.28.0
GenomicRanges_1.12.4 IRanges_1.18.1 BiocGenerics_0.6.0
loaded via a namespace (and not attached):
[1] biomaRt_2.16.0 bitops_1.0-6 edgeR_3.2.4 limma_3.16.8
RCurl_1.95-4.1 Rsamtools_1.12.4 stats4_3.0.0 tools_3.0.0
XML_3.98-1.1 zlibbioc_1.6.0
Hi Paolo,
too bad, you were almost there! When MEDIPS stopped processing due to
memory limitations, differential coverage was already calculated.
Unfortunately, the last step, i.e. creating the output result table,
was
too memory intensive for your set up. May I ask how many samples of
which
species do you process and what is the chosen window size? These are
the
parameters that influence memory requirements for the differential
coverage
analysis.
While I will investigate, how to further reduce memory requirements in
future versions, I see two immediate possible solutions without
reducing
window and sample sizes: i) migrate your analysis to a cluster with
more
memory, ii) analyze your data in chunks. While I understand that i) is
not
always available, this is what you can try for solution ii):
Update to MEDIPS version >= 1.11.16. This is currently available at
http://www.bioconductor.org/packages/2.13/bioc/html/MEDIPS.html and
will be
available as version 1.12.0 with the release of Bioconductor 2.13 on
October 15th 2013. In MEDIPS version >= 1.11.16, the function
MEDIPS.meth()
has the parameter "chr" which allows to process only a set of selected
chromosomes. In your case, I expect that it will be sufficient to
divide
the genome into two groups of chromosomes. Therefore, you can try
running
the MEDIPS.meth() function twice like:
res = MEDIPS.meth(MSet1 = MSets_groupA, MSet2 = MSets_groupB, chr =
chr_names(MSets_groupA[[1]])[1:x], ...)
resB = MEDIPS.meth(MSet1 = MSets_groupA, MSet2 = MSets_groupB, chr =
chr_names(MSets_groupA[[1]])[x+1:n], ...)
where n is the number of chromosomes included in your MEDIPS Sets and
x is
an arbitrary number that divides your chromosomes into two groups
(e.g.
round(n/2)). Please note that some modeled parameters (e.g. for
library
size normalization) will be slightly different for two subsets of the
data
compared to processing a genome wide table at once.
Finally, you can combine the result tables by
res = rbind(res, res_B)
rm(res_B)
and continue with a normal workflow.
All the best,
Lukas
On Wed, Oct 9, 2013 at 3:33 AM, Paolo Kunderfranco <
paolo.kunderfranco@gmail.com> wrote:
> Dear All,
> I am facing some problems with MEDIPS package,
> I Think I'am running out of memory once I run the command
MEDIPS.meth
> ................
> Adjusting p.values for multiple testing...
> Creating results table...
> Adding differential coverage results...
> Error: cannot allocate vector of size 207.9 Mb
> Inoltre: Warning messages:
> 1: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
> Reached total allocation of 24501Mb: see help(memory.size)
> 2: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
> Reached total allocation of 24501Mb: see help(memory.size)
> 3: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
> Reached total allocation of 24501Mb: see help(memory.size)
> 4: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
> Reached total allocation of 24501Mb: see help(memory.size)
>
> Is there a way to avoid this problem?
>
> I have a 24 Gb RAM workstation I don't see why it fails...
>
> Thanks,
> Paolo
>
>
> > sessionInfo()
> R version 3.0.0 (2013-04-03)
> Platform: x86_64-w64-mingw32/x64 (64-bit)
>
> locale:
> [1] LC_COLLATE=Italian_Italy.1252 LC_CTYPE=Italian_Italy.1252
> LC_MONETARY=Italian_Italy.1252 LC_NUMERIC=C
> LC_TIME=Italian_Italy.1252
>
> attached base packages:
> [1] parallel stats graphics grDevices utils datasets
> methods base
>
> other attached packages:
> [1] MEDIPS_1.10.0 BiocInstaller_1.10.3 gtools_3.1.0
> DNAcopy_1.34.0 BSgenome_1.28.0 Biostrings_2.28.0
> GenomicRanges_1.12.4 IRanges_1.18.1 BiocGenerics_0.6.0
>
> loaded via a namespace (and not attached):
> [1] biomaRt_2.16.0 bitops_1.0-6 edgeR_3.2.4 limma_3.16.8
> RCurl_1.95-4.1 Rsamtools_1.12.4 stats4_3.0.0 tools_3.0.0
> XML_3.98-1.1 zlibbioc_1.6.0
>
[[alternative HTML version deleted]]
Hi Lukas,
Specie is mouse, 4 samples plus 2 input, and ws=100.
The memory fails when I run MEDIPS.meth for differential analysis
between two samples and 1 input.
I bypassed the error as you suggested running 3 MEDIPS.meth with
chromosomes groups
Look forward for the new release!
Cheers,
Paolo
2013/10/9 Lukas Chavez <lukas.chavez.mailings at="" googlemail.com="">:
>
> Hi Paolo,
>
> too bad, you were almost there! When MEDIPS stopped processing due
to memory
> limitations, differential coverage was already calculated.
Unfortunately,
> the last step, i.e. creating the output result table, was too memory
> intensive for your set up. May I ask how many samples of which
species do
> you process and what is the chosen window size? These are the
parameters
> that influence memory requirements for the differential coverage
analysis.
>
> While I will investigate, how to further reduce memory requirements
in
> future versions, I see two immediate possible solutions without
reducing
> window and sample sizes: i) migrate your analysis to a cluster with
more
> memory, ii) analyze your data in chunks. While I understand that i)
is not
> always available, this is what you can try for solution ii):
>
> Update to MEDIPS version >= 1.11.16. This is currently available at
> http://www.bioconductor.org/packages/2.13/bioc/html/MEDIPS.html and
will be
> available as version 1.12.0 with the release of Bioconductor 2.13 on
October
> 15th 2013. In MEDIPS version >= 1.11.16, the function MEDIPS.meth()
has the
> parameter "chr" which allows to process only a set of selected
chromosomes.
> In your case, I expect that it will be sufficient to divide the
genome into
> two groups of chromosomes. Therefore, you can try running the
MEDIPS.meth()
> function twice like:
>
> res = MEDIPS.meth(MSet1 = MSets_groupA, MSet2 = MSets_groupB, chr =
> chr_names(MSets_groupA[[1]])[1:x], ...)
> resB = MEDIPS.meth(MSet1 = MSets_groupA, MSet2 = MSets_groupB, chr =
> chr_names(MSets_groupA[[1]])[x+1:n], ...)
>
> where n is the number of chromosomes included in your MEDIPS Sets
and x is
> an arbitrary number that divides your chromosomes into two groups
(e.g.
> round(n/2)). Please note that some modeled parameters (e.g. for
library size
> normalization) will be slightly different for two subsets of the
data
> compared to processing a genome wide table at once.
>
> Finally, you can combine the result tables by
>
> res = rbind(res, res_B)
> rm(res_B)
>
> and continue with a normal workflow.
>
> All the best,
> Lukas
>
>
>
>
>
>
> On Wed, Oct 9, 2013 at 3:33 AM, Paolo Kunderfranco
> <paolo.kunderfranco at="" gmail.com=""> wrote:
>>
>> Dear All,
>> I am facing some problems with MEDIPS package,
>> I Think I'am running out of memory once I run the command
MEDIPS.meth
>> ................
>> Adjusting p.values for multiple testing...
>> Creating results table...
>> Adding differential coverage results...
>> Error: cannot allocate vector of size 207.9 Mb
>> Inoltre: Warning messages:
>> 1: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
>> Reached total allocation of 24501Mb: see help(memory.size)
>> 2: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
>> Reached total allocation of 24501Mb: see help(memory.size)
>> 3: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
>> Reached total allocation of 24501Mb: see help(memory.size)
>> 4: In unlist(vlist, recursive = FALSE, use.names = FALSE) :
>> Reached total allocation of 24501Mb: see help(memory.size)
>>
>> Is there a way to avoid this problem?
>>
>> I have a 24 Gb RAM workstation I don't see why it fails...
>>
>> Thanks,
>> Paolo
>>
>>
>> > sessionInfo()
>> R version 3.0.0 (2013-04-03)
>> Platform: x86_64-w64-mingw32/x64 (64-bit)
>>
>> locale:
>> [1] LC_COLLATE=Italian_Italy.1252 LC_CTYPE=Italian_Italy.1252
>> LC_MONETARY=Italian_Italy.1252 LC_NUMERIC=C
>> LC_TIME=Italian_Italy.1252
>>
>> attached base packages:
>> [1] parallel stats graphics grDevices utils datasets
>> methods base
>>
>> other attached packages:
>> [1] MEDIPS_1.10.0 BiocInstaller_1.10.3 gtools_3.1.0
>> DNAcopy_1.34.0 BSgenome_1.28.0 Biostrings_2.28.0
>> GenomicRanges_1.12.4 IRanges_1.18.1 BiocGenerics_0.6.0
>>
>> loaded via a namespace (and not attached):
>> [1] biomaRt_2.16.0 bitops_1.0-6 edgeR_3.2.4
limma_3.16.8
>> RCurl_1.95-4.1 Rsamtools_1.12.4 stats4_3.0.0 tools_3.0.0
>> XML_3.98-1.1 zlibbioc_1.6.0
>
>