Question

segfault ReadAffy cause 'memory not mapped'

0

Entering edit mode

Brian D. Peyser PhD ▴ 70

@brian-d-peyser-phd-6106

Last seen 8.9 years ago

Below is some previous info on this issue; Dr Bolstad, did you issue a patch to affyio? I had just pulled the latest source from SVN with the hope that I could manage to hack it and noticed this at the top of read_abatch.c: ** Sept 18, 2013 - improve 64bit support for read_abatch I haven't tried running these big calcs since then. Should I give a try now? -Brian On Wed, 2013-08-21 at 16:21 -0700, Ben Bolstad wrote: > Well, internally read_abatch is using allocMatrix() to actually allocate > the main block of memory that will be used to store the probe intensities. > However, there are a lot of places where "int" is used as the indexing > variable. Probably if I had had better foresight when I wrote this code 10 > years ago, I'd have used something a bit more specific (eg int64_t). I'm > guessing it is one of these sorts of things that is causing the crash. > I'll try to get around to refactoring the code at some point. > > If you'd like you could send me the gdb backtrace at the point of the > segfault and I could investigate further. > > Best, > > Ben > > > > On 8/1/13 5:33 PM, Loraine, Ann wrote: > >> Hello, > >> > >> I am trying to process several thousand CEL files using the ReadAffy > >> command. > >> > >> The machine has 96 Gb RAM. > >> > >> However I get this error: > >> > >> > expr=ReadAffy(filenames=d.uniq$cel,celfile.path='CEL',sampleNam es=d.uniq$gsm,compress=T) > >> > >> *** caught segfault *** > >> address 0x7fc79b4b1048, cause 'memory not mapped' > >> > > > > I also have a problem loading many (3750) Affy hgu133plus2 arrays into > > an AffyBatch. I was able to run this with ~2900 arrays, but not since > > adding ~800 more. At right around 16 GiB allocated, I get a segfault > > like: > > > > *** caught segfault *** > > address 0x2aa6b6067048, cause 'memory not mapped' > > > > Traceback: > > 1: .Call("read_abatch", filenames, rm.mask, rm.outliers, rm.extra, > > ref.cdfName, dim.intensity[c(1, 2)], verbose, PACKAGE = "affyio") > > 2: read.affybatch(filenames = as.character(pdata$Filename)) > > <snip> -- Brian D. Peyser PhD Special Assistant to the Associate Director Office of the Associate Director Developmental Therapeutics Program Division of Cancer Treatment and Diagnosis National Cancer Institute National Institutes of Health 301-524-5587 (mobile)

Cancer hgu133plus2 affy PROcess Cancer hgu133plus2 affy PROcess • 1.4k views

ADD COMMENT • link updated 11.1 years ago by Ben Bolstad ★ 1.2k • written 11.1 years ago by Brian D. Peyser PhD ▴ 70

score 0 · Answer 1 · 2013-10-23

0

Entering edit mode

Ben Bolstad ★ 1.2k

@ben-bolstad-1494

Last seen 7.3 years ago

Those changes should be part of the most recent BioC release. Please let me know if you still discover any issues. Ben On Oct 23, 2013, at 7:07 AM, "Brian D. Peyser PhD" <brian.peyser at="" nih.gov=""> wrote: > Below is some previous info on this issue; Dr Bolstad, did you issue a > patch to affyio? > > I had just pulled the latest source from SVN with the hope that I could > manage to hack it and noticed this at the top of read_abatch.c: > > ** Sept 18, 2013 - improve 64bit support for read_abatch > > I haven't tried running these big calcs since then. Should I give a try > now? > > -Brian > > On Wed, 2013-08-21 at 16:21 -0700, Ben Bolstad wrote: >> Well, internally read_abatch is using allocMatrix() to actually allocate >> the main block of memory that will be used to store the probe intensities. >> However, there are a lot of places where "int" is used as the indexing >> variable. Probably if I had had better foresight when I wrote this code 10 >> years ago, I'd have used something a bit more specific (eg int64_t). I'm >> guessing it is one of these sorts of things that is causing the crash. >> I'll try to get around to refactoring the code at some point. >> >> If you'd like you could send me the gdb backtrace at the point of the >> segfault and I could investigate further. >> >> Best, >> >> Ben >> >> >>> On 8/1/13 5:33 PM, Loraine, Ann wrote: >>>> Hello, >>>> >>>> I am trying to process several thousand CEL files using the ReadAffy >>>> command. >>>> >>>> The machine has 96 Gb RAM. >>>> >>>> However I get this error: >>>> >>>>> expr=ReadAffy(filenames=d.uniq$cel,celfile.path='CEL',sampleName s=d.uniq$gsm,compress=T) >>>> >>>> *** caught segfault *** >>>> address 0x7fc79b4b1048, cause 'memory not mapped' >>>> >>> >>> I also have a problem loading many (3750) Affy hgu133plus2 arrays into >>> an AffyBatch. I was able to run this with ~2900 arrays, but not since >>> adding ~800 more. At right around 16 GiB allocated, I get a segfault >>> like: >>> >>> *** caught segfault *** >>> address 0x2aa6b6067048, cause 'memory not mapped' >>> >>> Traceback: >>> 1: .Call("read_abatch", filenames, rm.mask, rm.outliers, rm.extra, >>> ref.cdfName, dim.intensity[c(1, 2)], verbose, PACKAGE = "affyio") >>> 2: read.affybatch(filenames = as.character(pdata$Filename)) >>> > <snip> > > -- > Brian D. Peyser PhD > Special Assistant to the Associate Director > Office of the Associate Director > Developmental Therapeutics Program > Division of Cancer Treatment and Diagnosis > National Cancer Institute > National Institutes of Health > 301-524-5587 (mobile) >

ADD COMMENT • link 11.1 years ago Ben Bolstad ★ 1.2k

0

Entering edit mode

I checked my install and I have Bioconductor v2.13 running on R v3.0.2. However, when I get the latest affyio I get v1.30.0: > biocLite("affyio") BioC_mirror: http://bioconductor.org Using Bioconductor version 2.13 (BiocInstaller 1.12.0), R version 3.0.2. Installing package(s) 'affyio' trying URL 'http://bioconductor.org/packages/2.13/bioc/src/contrib/aff yio_1.30.0.tar.gz' <compiles> > citation("affyio") To cite package ?affyio? in publications use: Benjamin Milo Bolstad (). affyio: Tools for parsing Affymetrix data files. R package version 1.30.0. A BibTeX entry for LaTeX users is @Manual{, title = {affyio: Tools for parsing Affymetrix data files}, author = {Benjamin Milo Bolstad}, note = {R package version 1.30.0}, } ATTENTION: This citation information has been auto-generated from the package DESCRIPTION file and may need manual editing, see ?help("citation")?. Warning message: In citation("affyio") : no date field in DESCRIPTION file of package ?affyio? I looked at the DESCRIPTION file from the source I have and it says version is 1.31.0: Package: affyio Version: 1.31.0 Title: Tools for parsing Affymetrix data files Author: Benjamin Milo Bolstad <bmb at="" bmbolstad.com=""> Maintainer: Benjamin Milo Bolstad <bmb at="" bmbolstad.com=""> Depends: R (>= 2.6.0) Imports: zlibbioc Description: Routines for parsing Affymetrix data files based upon file format information. Primary focus is on accessing the CEL and CDF file formats. License: LGPL (>= 2) biocViews: Microarray, DataImport, Infrastructure LazyLoad: yes Do I need to run R-devel with Bioconductor 2.14 instead? I suppose I could just install affyio 1.31.0 from the SVN source. The 1.30.0 version does not contain the latest update, correct? Thanks! Brian On Wed, 2013-10-23 at 07:11 -0700, Benjamin Bolstad wrote: > Those changes should be part of the most recent BioC release. Please let me know if you still discover any issues. > > Ben > > > On Oct 23, 2013, at 7:07 AM, "Brian D. Peyser PhD" <brian.peyser at="" nih.gov=""> wrote: > > > Below is some previous info on this issue; Dr Bolstad, did you issue a > > patch to affyio? > > > > I had just pulled the latest source from SVN with the hope that I could > > manage to hack it and noticed this at the top of read_abatch.c: > > > > ** Sept 18, 2013 - improve 64bit support for read_abatch > > > > I haven't tried running these big calcs since then. Should I give a try > > now? > > > > -Brian > > > > On Wed, 2013-08-21 at 16:21 -0700, Ben Bolstad wrote: > >> Well, internally read_abatch is using allocMatrix() to actually allocate > >> the main block of memory that will be used to store the probe intensities. > >> However, there are a lot of places where "int" is used as the indexing > >> variable. Probably if I had had better foresight when I wrote this code 10 > >> years ago, I'd have used something a bit more specific (eg int64_t). I'm > >> guessing it is one of these sorts of things that is causing the crash. > >> I'll try to get around to refactoring the code at some point. > >> > >> If you'd like you could send me the gdb backtrace at the point of the > >> segfault and I could investigate further. > >> > >> Best, > >> > >> Ben > >> > >> > >>> On 8/1/13 5:33 PM, Loraine, Ann wrote: > >>>> Hello, > >>>> > >>>> I am trying to process several thousand CEL files using the ReadAffy > >>>> command. > >>>> > >>>> The machine has 96 Gb RAM. > >>>> > >>>> However I get this error: > >>>> > >>>>> expr=ReadAffy(filenames=d.uniq$cel,celfile.path='CEL',sampleNa mes=d.uniq$gsm,compress=T) > >>>> > >>>> *** caught segfault *** > >>>> address 0x7fc79b4b1048, cause 'memory not mapped' > >>>> > >>> > >>> I also have a problem loading many (3750) Affy hgu133plus2 arrays into > >>> an AffyBatch. I was able to run this with ~2900 arrays, but not since > >>> adding ~800 more. At right around 16 GiB allocated, I get a segfault > >>> like: > >>> > >>> *** caught segfault *** > >>> address 0x2aa6b6067048, cause 'memory not mapped' > >>> > >>> Traceback: > >>> 1: .Call("read_abatch", filenames, rm.mask, rm.outliers, rm.extra, > >>> ref.cdfName, dim.intensity[c(1, 2)], verbose, PACKAGE = "affyio") > >>> 2: read.affybatch(filenames = as.character(pdata$Filename)) > >>> > > <snip> > > > > -- > > Brian D. Peyser PhD > > Special Assistant to the Associate Director > > Office of the Associate Director > > Developmental Therapeutics Program > > Division of Cancer Treatment and Diagnosis > > National Cancer Institute > > National Institutes of Health > > 301-524-5587 (mobile) > > > -- Brian D. Peyser PhD Special Assistant to the Associate Director Office of the Associate Director Developmental Therapeutics Program Division of Cancer Treatment and Diagnosis National Cancer Institute National Institutes of Health 301-524-5587 (mobile)

ADD REPLY • link 11.1 years ago Brian D. Peyser PhD ▴ 70