Thanks Henrik, Herve for your suggestions. I will use the allocMatrix() method to detect the overflow: casting and multiplying the dimensions using the double type before comparison to INT_MAX. Cheers, Greg --- Gregoire Pau EMBL Research officer http://www.embl.de/~gpau/ On 09/04/2011 02:42, Hervé Pagès wrote: > Hi Greg, > > Seems like you have an integer overflow at the C level (src/io.c, > line 276): > > int width, height, nplanes; > > image = PROTECT(allocVector(REALSXP, width * height * nplanes)); > > For big values of width and/or height and/or nplanes, the result of > the product will be a negative integer. > > The way they catch integer multiplication overflow in the R base code > is with something like (found in src/main/array.c): > > if (nrow < 0 || ncol < 0) > error(_("negative extents to matrix")); > if ((double)nrow * (double)ncol > INT_MAX) > error(_("allocMatrix: too many elements specified")); > n = nrow * ncol; > PROTECT(s = allocVector(mode, n)); > > The code above is used in the allocMatrix() function. They also use > the same trick in alloc3DArray(): > > if (nrow < 0 || ncol < 0 || nface < 0) > error(_("negative extents to 3D array")); > if ((double)nrow * (double)ncol * (double)nface > INT_MAX) > error(_("alloc3Darray: too many elements specified")); > n = nrow * ncol * nface; > PROTECT(s = allocVector(mode, n)); > > Although this works fine in practice, this is not in theory 100% > portable (it relies on some subtle assumptions about the size of > the "int" type in C and how INT_MAX would accurately compare to a > a double). > > Another common technique is to use a type of integer that has at > least twice the size of the "int" type. Note that, according to > the ANSI C specs, the "long int" type is not guaranteed to satisfy > this. Not even the "long long int". But it seems that in practice, > and probably on most (if not all) platforms where R is supported, > "int" is 32-bit and "long long int" is 64-bit. So the following > should be reasonably portable: > > int width, height, nplanes; > unsigned long long n; > > n = width; > n *= height; > if (n > INT_MAX) > error("readCellomics: image is too big"); > n *= nplanes; > if (n > INT_MAX) > error("readCellomics: image is too big") > image = PROTECT(allocVector(REALSXP, n)); > > The advantage of this solution over the "double" solution above is > that the multiplication is performed only once. > > Note that INT_MAX is defined in limits.h > > A 3rd solution which seems to be cleaner is to just use unsigned > int. Since "modulo arithmetic" is well defined on unsigned int > (according to the C99 standard), one can do: > > int width, height, nplanes, n; > unsigned int uwidth, uheight, unplanes, n1, n2; > > uwidth = width; > uheight = height; > n1 = uwidth * uheight; > if ((uheight != 0U) && (n1 / uheight != uwidth)) > error("readCellomics: image is too big"); > unplanes = nplanes; > n2 = n1 * unplanes; > if ((unplanes != 0U) && (n2 / unplanes != n1)) > error("readCellomics: image is too big"); > n = n2; > image = PROTECT(allocVector(REALSXP, n)); > > But why not just use alloc3DArray() or allocMatrix() in your code in > the first place. They will not only catch the integer overflow problem > for you, they will also set the dim attributes on 'image'. > > Hope this helps, > > H. > > > On 11-04-08 12:17 AM, Gregoire Pau wrote: >> This is right, Henrik. >> >> The image is 52703x20688: this is huge and I cannot display it on my >> computer. This image would take 52703x20688x8 bytes in memory, i.e. >> about 8 GBytes. I am not sure if R (even the 64-bit version of R) can >> handle such big objects. >> >> Greg >> --- >> Gregoire Pau >> EMBL Research Officer >> http://www.embl.de/~gpau/ >> >> On 07/04/11 18:50, Vladimir Morozov wrote: >>> Henrik, >>> >>> I think you make the right guess. >>> >>> Anyway the tiff file is >>> 0500101820.0.tif - >>> https://docs.google.com/leaf?id=0B2NLCeM3RRN2ODIxYWVmZjItYzYwNS00Y jJjLWE2YjQtYjQwZGUzZTJkOTBi&hl=en >>> >>> >>> >>> Thanks, >>> >>> Vlad >>> >>> >>> -----Original Message----- >>> From: henrik.bengtsson at gmail.com [mailto:henrik.bengtsson at gmail.com] >>> On Behalf Of Henrik Bengtsson >>> Sent: Thursday, April 07, 2011 12:46 PM >>> To: Vladimir Morozov >>> Cc: Gregoire Pau; bioconductor at stat.math.ethz.ch >>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages", >>> as.character(files), as.integer(colormode), >>> >>> A guess: Your image dimension might be too large. In R, >>> matrices/arrays are stored as vectors with a dimension attribute >>> meaning the number of elements is limited by prod(dim(x)) for an array >>> 'x', not max(dim(x)). Since indices are 32-bit integers in R, you may >>> be hitting the maximum limit: >>> >>>> .Machine$integer.max >>> [1] 2147483647 >>> >>> # For your small image that works >>>> dim<- c(26352,10344,3) >>>> prod(dim) >>> [1] 817755264 >>>> (prod(dim)<= .Machine$integer.max) >>> [1] TRUE >>> >>> # For the 90/50=1.8 times larger one >>>> dim<- 1.8*c(26352,10344,3) >>>> prod(dim) >>> [1] 4769148700 >>>> (prod(dim)<= .Machine$integer.max) >>> [1] FALSE >>> >>> /Henrik >>> >>> >>> On Thu, Apr 7, 2011 at 9:23 AM, Vladimir Morozov<vmorozov at="" als.net=""> >>> wrote: >>>> Hi Gregory, >>>> >>>> It dosen't seem to be multiple version problem: >>>> >>>> [rstats:ImageAnal] ldd >>>> /usr/local/lib64/R/library/EBImage/libs/EBImage.so|grep -i magic >>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3 >>>> (0x00007f264092f000) >>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3 >>>> (0x00007f263e62c000) [rstats:ImageAnal] which convert /usr/bin/convert >>>> [rstats:ImageAnal] ldd /usr/bin/convert >>>> linux-vdso.so.1 => (0x00007fff09bff000) >>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3 >>>> (0x00007fdad3f45000) >>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3 >>>> (0x00007fdad3c26000) >>>> .... >>>> >>>> >>>> I can resize the tiff image with ImageMagic: >>>> convert -channel B -resize '50%' >>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>> /data/projects/ImageAnal/tiff/0500101820.05.png >>>> And then get it into R : >>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.05.png') >>>>> dim(x) >>>> [1] 26352 10344 3 >>>> >>>> However bigger resolution images fail: >>>> convert -channel B -resize '90%' >>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>> /data/projects/ImageAnal/tiff/0500101820.09.png >>>> >>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.09.png') >>>> Error in .Call("lib_readImages", as.character(files), >>>> as.integer(colormode), : >>>> negative length vectors are not allowed >>>> >>>> So it doesn't look as ImageMagic problem. If you still want the tiff >>>> file, I can post it >>>> >>>> Best >>>> Vlad >>>> >>>> >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Gregoire Pau [mailto:gregoire.pau at embl.de] >>>> Sent: Thursday, April 07, 2011 11:53 AM >>>> To: Vladimir Morozov >>>> Cc: 'bioconductor at stat.math.ethz.ch' >>>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages", >>>> as.character(files), as.integer(colormode), >>>> >>>> Hello Vladimir, >>>> >>>> If ImageMagick is able to read the image, EBImage should be able to >>>> do so. Do you have several installed versions of ImageMagick ? To >>>> which ImageMagick version EBImage is linked to (try otool or ldd on >>>> EBImage.so or EBImage.dll) ? >>>> >>>> TIFF is a not a file format but a file container which can contain >>>> exotic formats: could you please send me your image for inspection ? >>>> >>>> Cheers, >>>> >>>> Greg >>>> --- >>>> Gregoire Pau >>>> EMBL Research Officer >>>> http://www.embl.de/~gpau/ >>>> >>>> On 07/04/11 16:48, Vladimir Morozov wrote: >>>>> Hi, >>>>> >>>>> I get image reading error: >>>>> >>>>>> x<- readImage( >>>>> >>>>> "/data/projects/ImageAnal/tiff/0500101820.0.tif" >>>>> >>>>> ) >>>>> >>>>> Error in .Call("lib_readImages", as.character(files), >>>>> as.integer(colormode), : >>>>> >>>>> negative length vectors are not allowed >>>>> >>>>>> packageVersion('EBImage') >>>>> >>>>> [1] '3.6.0' >>>>> >>>>>> >>>>> >>>>> ImageMagic can work with this image: >>>>> >>>>> convert -channel B -resize '70%' >>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>>> /data/projects/ImageAnal/tiff/0500101820.07.jpeg& >>>>> >>>>> >>>>> >>>>> Tthis image is the bigest image from SVS stack: >>>>> >>>>> convert /data/projects/ImageAnal/svs/0500101820.svs >>>>> /data/projects/ImageAnal/tiff/0500101820.%d.tif >>>>> >>>>> [rstats:ImageAnal] ll -h /data/projects/ImageAnal/tiff/0500101820.* >>>>> -rw-r--r-- 1 vmorozov users 24K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.1.tif >>>>> -rw-r--r-- 1 vmorozov users 81M 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>>> -rw-r--r-- 1 vmorozov users 155K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.5.tif >>>>> -rw-r--r-- 1 vmorozov users 49K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.4.tif >>>>> -rw-r--r-- 1 vmorozov users 196K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.3.tif >>>>> -rw-r--r-- 1 vmorozov users 3.0M 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.2.tif >>>>> [rstats:ImageAnal] convert -v >>>>> Version: ImageMagick 6.6.1-0 2010-07-29 Q16 >>>>> http://www.imagemagick.org >>>>> Copyright: Copyright (C) 1999-2010 ImageMagick Studio LLC >>>>> Features: OpenMP >>>>> ..... >>>>> >>>>> >>>>> >>>>> I have the same problem with other SVS stacks, i.e readImage fails to >>>>> read the most quality images. it works fine on smaller images >>>>> >>>>> >>>>> >>>>> Any suggestions would be apriciated >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> Vladimir Morozov >>>>> Sr. Computational Biologist >>>>> ALS Therapy Development Institute >>>>> 215 First Street, Cambridge MA, 02142 www.als.net<http: www.als.net=""> >>>>> Help End ALS! Click Here<http: als.net="" endals.aspx=""> or Text >>>>> "EndALS" to 20222 to make a $5 donation towards research. >>>>> >>>>> >>>>> >>>>> ********************************************************************* >>>>> * >>>>> ***************** The information contained in this electronic >>>>> message is ...{{dropped:21}} >>>>> >>>>> _______________________________________________ >>>>> Bioconductor mailing list >>>>> Bioconductor at r-project.org >>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>> Search the archives: >>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>> >>>> >>>> ********************************************************************** >>>> ***************** The information contained in this electronic message >>>> is ...{{dropped:18}} >>>> >>>> _______________________________________________ >>>> Bioconductor mailing list >>>> Bioconductor at r-project.org >>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>> >>> >>> >>> ****************************************************************** ********************* >>> >>> >>> The information contained in this electronic message i...{{dropped:3}} >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor > >

On 11-04-08 05:42 PM, Hervé Pagès wrote: [...] > A 3rd solution which seems to be cleaner is to just use unsigned > int. Since "modulo arithmetic" is well defined on unsigned int > (according to the C99 standard), one can do: > > int width, height, nplanes, n; > unsigned int uwidth, uheight, unplanes, n1, n2; > > uwidth = width; > uheight = height; > n1 = uwidth * uheight; > if ((uheight != 0U) && (n1 / uheight != uwidth)) > error("readCellomics: image is too big"); > unplanes = nplanes; > n2 = n1 * unplanes; > if ((unplanes != 0U) && (n2 / unplanes != n1)) > error("readCellomics: image is too big"); > n = n2; Oops, this is incorrect, sorry. n2 should be compared to INT_MAX before it is assigned to n. H. > image = PROTECT(allocVector(REALSXP, n)); > > But why not just use alloc3DArray() or allocMatrix() in your code in > the first place. They will not only catch the integer overflow problem > for you, they will also set the dim attributes on 'image'. > > Hope this helps, > > H. > > > On 11-04-08 12:17 AM, Gregoire Pau wrote: >> This is right, Henrik. >> >> The image is 52703x20688: this is huge and I cannot display it on my >> computer. This image would take 52703x20688x8 bytes in memory, i.e. >> about 8 GBytes. I am not sure if R (even the 64-bit version of R) can >> handle such big objects. >> >> Greg >> --- >> Gregoire Pau >> EMBL Research Officer >> http://www.embl.de/~gpau/ >> >> On 07/04/11 18:50, Vladimir Morozov wrote: >>> Henrik, >>> >>> I think you make the right guess. >>> >>> Anyway the tiff file is >>> 0500101820.0.tif - >>> https://docs.google.com/leaf?id=0B2NLCeM3RRN2ODIxYWVmZjItYzYwNS00Y jJjLWE2YjQtYjQwZGUzZTJkOTBi&hl=en >>> >>> >>> >>> Thanks, >>> >>> Vlad >>> >>> >>> -----Original Message----- >>> From: henrik.bengtsson at gmail.com [mailto:henrik.bengtsson at gmail.com] >>> On Behalf Of Henrik Bengtsson >>> Sent: Thursday, April 07, 2011 12:46 PM >>> To: Vladimir Morozov >>> Cc: Gregoire Pau; bioconductor at stat.math.ethz.ch >>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages", >>> as.character(files), as.integer(colormode), >>> >>> A guess: Your image dimension might be too large. In R, >>> matrices/arrays are stored as vectors with a dimension attribute >>> meaning the number of elements is limited by prod(dim(x)) for an array >>> 'x', not max(dim(x)). Since indices are 32-bit integers in R, you may >>> be hitting the maximum limit: >>> >>>> .Machine$integer.max >>> [1] 2147483647 >>> >>> # For your small image that works >>>> dim<- c(26352,10344,3) >>>> prod(dim) >>> [1] 817755264 >>>> (prod(dim)<= .Machine$integer.max) >>> [1] TRUE >>> >>> # For the 90/50=1.8 times larger one >>>> dim<- 1.8*c(26352,10344,3) >>>> prod(dim) >>> [1] 4769148700 >>>> (prod(dim)<= .Machine$integer.max) >>> [1] FALSE >>> >>> /Henrik >>> >>> >>> On Thu, Apr 7, 2011 at 9:23 AM, Vladimir Morozov<vmorozov at="" als.net=""> >>> wrote: >>>> Hi Gregory, >>>> >>>> It dosen't seem to be multiple version problem: >>>> >>>> [rstats:ImageAnal] ldd >>>> /usr/local/lib64/R/library/EBImage/libs/EBImage.so|grep -i magic >>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3 >>>> (0x00007f264092f000) >>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3 >>>> (0x00007f263e62c000) [rstats:ImageAnal] which convert /usr/bin/convert >>>> [rstats:ImageAnal] ldd /usr/bin/convert >>>> linux-vdso.so.1 => (0x00007fff09bff000) >>>> libMagickCore.so.3 => /usr/lib64/libMagickCore.so.3 >>>> (0x00007fdad3f45000) >>>> libMagickWand.so.3 => /usr/lib64/libMagickWand.so.3 >>>> (0x00007fdad3c26000) >>>> .... >>>> >>>> >>>> I can resize the tiff image with ImageMagic: >>>> convert -channel B -resize '50%' >>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>> /data/projects/ImageAnal/tiff/0500101820.05.png >>>> And then get it into R : >>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.05.png') >>>>> dim(x) >>>> [1] 26352 10344 3 >>>> >>>> However bigger resolution images fail: >>>> convert -channel B -resize '90%' >>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>> /data/projects/ImageAnal/tiff/0500101820.09.png >>>> >>>>> x<- readImage('/data/projects/ImageAnal/tiff/0500101820.09.png') >>>> Error in .Call("lib_readImages", as.character(files), >>>> as.integer(colormode), : >>>> negative length vectors are not allowed >>>> >>>> So it doesn't look as ImageMagic problem. If you still want the tiff >>>> file, I can post it >>>> >>>> Best >>>> Vlad >>>> >>>> >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Gregoire Pau [mailto:gregoire.pau at embl.de] >>>> Sent: Thursday, April 07, 2011 11:53 AM >>>> To: Vladimir Morozov >>>> Cc: 'bioconductor at stat.math.ethz.ch' >>>> Subject: Re: [BioC] EBImage Error in .Call("lib_readImages", >>>> as.character(files), as.integer(colormode), >>>> >>>> Hello Vladimir, >>>> >>>> If ImageMagick is able to read the image, EBImage should be able to >>>> do so. Do you have several installed versions of ImageMagick ? To >>>> which ImageMagick version EBImage is linked to (try otool or ldd on >>>> EBImage.so or EBImage.dll) ? >>>> >>>> TIFF is a not a file format but a file container which can contain >>>> exotic formats: could you please send me your image for inspection ? >>>> >>>> Cheers, >>>> >>>> Greg >>>> --- >>>> Gregoire Pau >>>> EMBL Research Officer >>>> http://www.embl.de/~gpau/ >>>> >>>> On 07/04/11 16:48, Vladimir Morozov wrote: >>>>> Hi, >>>>> >>>>> I get image reading error: >>>>> >>>>>> x<- readImage( >>>>> >>>>> "/data/projects/ImageAnal/tiff/0500101820.0.tif" >>>>> >>>>> ) >>>>> >>>>> Error in .Call("lib_readImages", as.character(files), >>>>> as.integer(colormode), : >>>>> >>>>> negative length vectors are not allowed >>>>> >>>>>> packageVersion('EBImage') >>>>> >>>>> [1] '3.6.0' >>>>> >>>>>> >>>>> >>>>> ImageMagic can work with this image: >>>>> >>>>> convert -channel B -resize '70%' >>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>>> /data/projects/ImageAnal/tiff/0500101820.07.jpeg& >>>>> >>>>> >>>>> >>>>> Tthis image is the bigest image from SVS stack: >>>>> >>>>> convert /data/projects/ImageAnal/svs/0500101820.svs >>>>> /data/projects/ImageAnal/tiff/0500101820.%d.tif >>>>> >>>>> [rstats:ImageAnal] ll -h /data/projects/ImageAnal/tiff/0500101820.* >>>>> -rw-r--r-- 1 vmorozov users 24K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.1.tif >>>>> -rw-r--r-- 1 vmorozov users 81M 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.0.tif >>>>> -rw-r--r-- 1 vmorozov users 155K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.5.tif >>>>> -rw-r--r-- 1 vmorozov users 49K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.4.tif >>>>> -rw-r--r-- 1 vmorozov users 196K 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.3.tif >>>>> -rw-r--r-- 1 vmorozov users 3.0M 2011-04-06 15:18 >>>>> /data/projects/ImageAnal/tiff/0500101820.2.tif >>>>> [rstats:ImageAnal] convert -v >>>>> Version: ImageMagick 6.6.1-0 2010-07-29 Q16 >>>>> http://www.imagemagick.org >>>>> Copyright: Copyright (C) 1999-2010 ImageMagick Studio LLC >>>>> Features: OpenMP >>>>> ..... >>>>> >>>>> >>>>> >>>>> I have the same problem with other SVS stacks, i.e readImage fails to >>>>> read the most quality images. it works fine on smaller images >>>>> >>>>> >>>>> >>>>> Any suggestions would be apriciated >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> Vladimir Morozov >>>>> Sr. Computational Biologist >>>>> ALS Therapy Development Institute >>>>> 215 First Street, Cambridge MA, 02142 www.als.net<http: www.als.net=""> >>>>> Help End ALS! Click Here<http: als.net="" endals.aspx=""> or Text >>>>> "EndALS" to 20222 to make a $5 donation towards research. >>>>> >>>>> >>>>> >>>>> ********************************************************************* >>>>> * >>>>> ***************** The information contained in this electronic >>>>> message is ...{{dropped:21}} >>>>> >>>>> _______________________________________________ >>>>> Bioconductor mailing list >>>>> Bioconductor at r-project.org >>>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>>> Search the archives: >>>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>> >>>> >>>> ********************************************************************** >>>> ***************** The information contained in this electronic message >>>> is ...{{dropped:18}} >>>> >>>> _______________________________________________ >>>> Bioconductor mailing list >>>> Bioconductor at r-project.org >>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>>> >>> >>> >>> ****************************************************************** ********************* >>> >>> >>> The information contained in this electronic message i...{{dropped:3}} >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor > > -- Hervé Pagès Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M2-B876 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpages at fhcrc.org Phone: (206) 667-5791 Fax: (206) 667-1319