I have a couple of questions about estimation of dispersion in DeSeq2 vs. DeSeq. The paper and manual on DeSeq and DeSeq2 are very clear but I have a hard time understanding what is the ‘correct’ or ‘recommended’ method of estimating the dispersion is for RNA-Seq. Does this depend on the sample size? Are there other assumptions that have to be met for each dispersion estimate type (mean, local or parametric)? is there a particular reason behind ‘tag-wise’ estimation not being in DeSeq2?

Thank you!

Hi JK,

For DESeq2, we have 3 options for the type of trend line: fitType = "parametric", "local" or "mean". These are described in ?estimateDispersions, but roughly the recommendation is: for decreasing gene-wise dispersion estimates over mean (using plotDispEsts) one should use parametric, unless the parametric fitting procedure does not work, in which case use "local" (local regression is actually automatically substituted with a message in the case that the parametric fitting procedure does not converge.) The "mean" option is useful when there is no apparent dependence of dispersion estimates over mean (using plotDispEsts). This choice does not depend on sample size, but on the apparent dependence of the gene-wise estimates (the MLE for each gene) on the mean of counts. 

If you are referring to the the tagwise estimation in edgeR, the tagwise estimation is similar to the estimateDispersionsMAP() step in DESeq2, which is the last step in estimateDispersions(), which is automatically used by DESeq().