Well spotted. So I guess the authors of mt.maxT() were clever enough to note that complete enumeration is possible for small number of iterations and wrote an overriding condition (using mt.getmaxB) as not waste time with 10000 iteration.ns. I guess another way of looking at your problem is that 3 vs. 3 is does not have sufficient power to detect with this correction and the obtained p-values reflect this. One possibility is to use the normal t-test or wilcoxon to obtain a vector of p-values and then adjust it for multiple testing. However since I am no expert in multiple hypothesis testing, I hope someone else can answer your question. Sorry. Regards, Adai On Mon, 2005-02-14 at 11:25 -0500, Straubhaar, Juerg wrote: > Hi Adai, > > The multtest version I am using is version 1.5.2 and the default value of B is 10000. Even if I set B to 100000 only 20 permutations will be calculated. I have two classes and 3 members in each class. The maximum number of permutations of the 6 columns (2 X 3) is 20. So how can I do 10000 permutations? > > Juerg > > > -----Original Message----- > From: Adaikalavan Ramasamy [mailto:ramasamy@cancer.org.uk] > Sent: Sun 2/13/2005 10:45 PM > To: Straubhaar, Juerg > Cc: BioConductor mailing list > Subject: Re: [BioC] questions on results from mt.maxT (multtest) > > The resolution of your p-value by permutation test is 1/B where B is the > number of iterations. In the output shows, I think it says B=20. Thus > your resolution (or smallest p-value) is 1/20 = 0.05. > > However I am wondering why did your function mt.maxT() default to B=20 > when the current of multtest (1.5.2) has a default of B=10000 ? > > Are you by any chance using a very old version of multtest ? If so, you > can try upgrading or explicitly set B=10000 (or something higher) in > mt.maxT. Otherwise check your inputs. > > Regards, Adai > > > On Sun, 2005-02-13 at 13:02 -0500, Straubhaar, Juerg wrote: > > Im am using R version 2.0.1, multtest version 1.5.2. > > > > Microarray experiment: wildtype tissue vs KO tissue; affy MOE430_2 > > Experiments in triplicates. > > > > ExprSet@exprs matrix has dimension: 45101 6 > > > > I used nonspecific filtering which reduces the number of genes to 1815 (exprSub) > > > > I am using multtest to get a set of differentially expressed genes: > > > > resT <-mt.maxT(exprSub@exprs, c(0,1,0,1,0)) > > This runs 20 permutations (complete enumeration) > > > > sort(resT$rawp) looks like this: > > > > [1] 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 > > [20] 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 > > [39] 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 > > ... > > [1711] 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 > > [1729] 0.9 0.9 0.9 0.9 0.9 0.9 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 > > [1747] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 > > [1765] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 > > [1783] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 > > [1801] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 > > > > It looks like these p-values are rounded to one decimal digit. From these results none of the 1815 genes are differentially expressed between WT and KO, which is not quite believable. I would like to get p-values precise to several digits. Maybe there is something else I don't understand. > > > > getOption('digits') 7 > > > > Thank you. > > > > Juerg Straubhaar > > Umass Med > > > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor@stat.math.ethz.ch > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > > >

Hi Perry, in fact Wilcoxon-test will suffer from similar granularity as a permutation test. A straight-forward t-test is also problematic, due to the small number of replicates and the resulting high variance of the denominator in the t-test. My advice would be to use a moderated t-test such as available through the eBayes function in limma, well-described in the vignette, or in fact, the user's guide. Best wishes Wolfgang P.D. Moerland wrote: > Hi Juerg, > > what Wolfgang means is that in your case with six samples, three from > each class, you can only have (6 choose 3)= 20 permutations of the class > labels. That's why your p-values will always be multiples of 1/20. A > permutation test is not suitable with so few samples; you should use one > of the standard tests (t, Wilcoxon, limma's robust t) and then correct > for mulitple testing. > > Best wishes, > Perry > > ----- Original Message ----- > From: "Straubhaar, Juerg" <juerg.straubhaar@umassmed.edu> > Date: Monday, February 14, 2005 5:08 pm > Subject: RE: [BioC] questions on results from mt.maxT (multtest) > > >>Hi Wolfgang, >> >>I don't quite understand what you mean by 'parametric test': the >>mt.maxT function has a parameter 'nonpara' which is set by default >>to 'n'. I assume a parametric test was performed. >> >>Juerg >> >> >>-----Original Message----- >>From: Wolfgang Huber [mailto:huber@ebi.ac.uk] >>Sent: Sun 2/13/2005 12:12 PM >>To: Straubhaar, Juerg >>Cc: bioconductor@stat.math.ethz.ch >>Subject: Re: [BioC] questions on results from mt.maxT (multtest) >> >>Hi Juerg, >> >>the p-values from a permutation test are multiples of 1/(number of >>permutations). If you want p-values with more digits, you need to >>do a >>parametric test. >> >>You say your exprs matrix has 6 colums, but in the call to mt.maxT >>you >>only have 5 classlabels. Wrong? >> >> Bw >> Wolfgang >> >> >>Straubhaar, Juerg wrote: >> >>>Im am using R version 2.0.1, multtest version 1.5.2. >>> >>>Microarray experiment: wildtype tissue vs KO tissue; affy MOE430_2 >>>Experiments in triplicates. >>> >>>ExprSet@exprs matrix has dimension: 45101 6 >>> >>>I used nonspecific filtering which reduces the number of genes to >> >>1815 (exprSub) >> >>>I am using multtest to get a set of differentially expressed genes: >>> >>>resT <-mt.maxT(exprSub@exprs, c(0,1,0,1,0)) >>>This runs 20 permutations (complete enumeration) >>> >>>sort(resT$rawp) looks like this: >>> >>>[1] 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 >> >>0.1 0.1 0.1 0.1 >> >>>[20] 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 >> >>0.1 0.1 0.1 0.1 >> >>>[39] 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 >>>... >>>[1711] 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 >> >>0.9 0.9 0.9 0.9 >> >>>[1729] 0.9 0.9 0.9 0.9 0.9 0.9 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >> >>1.0 1.0 1.0 1.0 >> >>>[1747] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >> >>1.0 1.0 1.0 1.0 >> >>>[1765] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >> >>1.0 1.0 1.0 1.0 >> >>>[1783] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >> >>1.0 1.0 1.0 1.0 >> >>>[1801] 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >>> >>>It looks like these p-values are rounded to one decimal digit. >> >>From these results none of the 1815 genes are differentially >>expressed between WT and KO, which is not quite believable. I would >>like to get p-values precise to several digits. Maybe there is >>something else I don't understand. >> >>>getOption('digits') 7 >>> >>>Thank you. >>> >>>Juerg Straubhaar >>>Umass Med >>> >>>_______________________________________________ >>>Bioconductor mailing list >>>Bioconductor@stat.math.ethz.ch >>>https://stat.ethz.ch/mailman/listinfo/bioconductor >> >> >>-- >>Best regards >> Wolfgang >> >>------------------------------------- >>Wolfgang Huber >>European Bioinformatics Institute >>European Molecular Biology Laboratory >>Cambridge CB10 1SD >>England >>Phone: +44 1223 494642 >>Fax: +44 1223 494486 >>Http: www.ebi.ac.uk/huber >> >>_______________________________________________ >>Bioconductor mailing list >>Bioconductor@stat.math.ethz.ch >>https://stat.ethz.ch/mailman/listinfo/bioconductor >> > > > _______________________________________________ > Bioconductor mailing list > Bioconductor@stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor -- Best regards Wolfgang ------------------------------------- Wolfgang Huber European Bioinformatics Institute European Molecular Biology Laboratory Cambridge CB10 1SD England Phone: +44 1223 494642 Fax: +44 1223 494486 Http: www.ebi.ac.uk/huber