At 08:00 PM 16/06/2006, bioconductor-request at stat.math.ethz.ch wrote: >Date: Fri, 16 Jun 2006 11:30:18 +0200 >From: Pedro L?pez Romero <plopez at="" cnic.es=""> >Subject: [BioC] decideTests with nestedF >To: <bioconductor at="" stat.math.ethz.ch=""> >Content-Type: text/plain > >Dear list, > > I have an experiment with several contrasts associated to different >experimental conditions for each gene, so I want to select genes in a first >step by their moderated-F statistic in order to reduce the number of genes >for multiple contrasts across genes. > >For this purpose, I am using decideTests with nestedF, as it is said in >limma user guide (chapter 10, p 51). However, when I use decideTests with >method="nestedF", I get quite a few genes that have an adjusted p value in >their separate t-tests contrast greater than 0.9. These genes have a really >small M value as to accept that they are really differentially expressed >between conditions. Does this make sense or I am doing something wrong?.- > >The adjusted pvalue that I am talking about is the one that I see after >applying the topTable function for the corresponding single contrast between >two experimental conditions. In this case, the adj.P.Val is the same as the >ones in decideTests with method ="separate", since we are applying the same >tests. > >My code is: > >decideTests(fit2,method="nestedF",adjust.method="fdr",p.value=0,05) >topTable(fit2,coef=i,number=100 adjust="fdr") > >Would be the function p.adjust(fit2$F.p.value,method="BH") (as Gordon >proposed in a recent email about this topic) more appropriate in order to do >what I what to do? > >Any comment will be of great help to me. > >Thanks a lot.- >p.- Dear Pedro, Different adjustment methods do give different results. It is as simple as that. The phenomenon that you have observed is perfectly possible and quite intentional. (Surely this is not surprising. Afterall, if different adjustment methods gave the same result, then there wouldn't be different methods.) Please understand that nestedF does not give any formal control of FDR at the contrast level. The only control is at the probe level: by the above code, you have controlled the expected proportion of genes falsely identified to have *any* significant contrasts to be less than 0.05. If you are seeking control of FDR at the contrast level, you should probably use another adjustment method. The methods "global" and "separate" can both give control of FDR at the contrast level. The strategy which you seem to want to do, using F-tests first and then t-tests for the significant genes, with control of FWER or FDR at each level, is called method="heirarchical" in limma. Please understand also that control of FDR behaves differently from control of the family-wise error rate (FWER), which is the traditional emphasis of multiple testing. The idea of hierarchical testing with F-tests before t-tests is far more effective for controlling FWER than it is for controlling FDR. It doesn't make much difference for FDR. The real motivation for nestedF is to discover genes which respond simultaneously to several contrasts. You will find that the gene which is worrying you so much has several significant contrasts. nestedF is designed to become less and less rigorous as it works down the contrasts for a given gene. As I said before, this is quite intentional. It produceds a method which is better at finding simultaneously contrasts but worse at finding genes which respond to only one contrast. Hope this helps. Best wishes Gordon

Hi, Thank you very much to all of you for the many comments and suggestions. They have been of great help and I think that I understand much better now the differences between the different methods.- I have compared the genes that I have obtained using *separate*, *nestedF* and *hierarchical* and there is a lot of agreement. Of course, there are some genes that are specifically selected by *nestedF* that have a high p-value in the *separate* list, but now I think that I understand why. Again, thank you very much.- Pedro.- -----Mensaje original----- De: Gordon Smyth [mailto:smyth at wehi.EDU.AU] Enviado el: s?bado, 17 de junio de 2006 6:01 Para: Pedro L?pez Romero CC: bioconductor at stat.math.ethz.ch Asunto: [BioC] decideTests with nestedF At 08:00 PM 16/06/2006, bioconductor-request at stat.math.ethz.ch wrote: >Date: Fri, 16 Jun 2006 11:30:18 +0200 >From: Pedro L?pez Romero <plopez at="" cnic.es=""> >Subject: [BioC] decideTests with nestedF >To: <bioconductor at="" stat.math.ethz.ch=""> >Content-Type: text/plain > >Dear list, > > I have an experiment with several contrasts associated to different >experimental conditions for each gene, so I want to select genes in a first >step by their moderated-F statistic in order to reduce the number of genes >for multiple contrasts across genes. > >For this purpose, I am using decideTests with nestedF, as it is said in >limma user guide (chapter 10, p 51). However, when I use decideTests with >method="nestedF", I get quite a few genes that have an adjusted p value in >their separate t-tests contrast greater than 0.9. These genes have a really >small M value as to accept that they are really differentially expressed >between conditions. Does this make sense or I am doing something wrong?.- > >The adjusted pvalue that I am talking about is the one that I see after >applying the topTable function for the corresponding single contrast between >two experimental conditions. In this case, the adj.P.Val is the same as the >ones in decideTests with method ="separate", since we are applying the same >tests. > >My code is: > >decideTests(fit2,method="nestedF",adjust.method="fdr",p.value=0,05) >topTable(fit2,coef=i,number=100 adjust="fdr") > >Would be the function p.adjust(fit2$F.p.value,method="BH") (as Gordon >proposed in a recent email about this topic) more appropriate in order to do >what I what to do? > >Any comment will be of great help to me. > >Thanks a lot.- >p.- Dear Pedro, Different adjustment methods do give different results. It is as simple as that. The phenomenon that you have observed is perfectly possible and quite intentional. (Surely this is not surprising. Afterall, if different adjustment methods gave the same result, then there wouldn't be different methods.) Please understand that nestedF does not give any formal control of FDR at the contrast level. The only control is at the probe level: by the above code, you have controlled the expected proportion of genes falsely identified to have *any* significant contrasts to be less than 0.05. If you are seeking control of FDR at the contrast level, you should probably use another adjustment method. The methods "global" and "separate" can both give control of FDR at the contrast level. The strategy which you seem to want to do, using F-tests first and then t-tests for the significant genes, with control of FWER or FDR at each level, is called method="heirarchical" in limma. Please understand also that control of FDR behaves differently from control of the family-wise error rate (FWER), which is the traditional emphasis of multiple testing. The idea of hierarchical testing with F-tests before t-tests is far more effective for controlling FWER than it is for controlling FDR. It doesn't make much difference for FDR. The real motivation for nestedF is to discover genes which respond simultaneously to several contrasts. You will find that the gene which is worrying you so much has several significant contrasts. nestedF is designed to become less and less rigorous as it works down the contrasts for a given gene. As I said before, this is quite intentional. It produceds a method which is better at finding simultaneously contrasts but worse at finding genes which respond to only one contrast. Hope this helps. Best wishes Gordon