Hi,

We would like to compare the expression variation across mulitple groups, e.g. to examine whether single cell samples in 10 hours (n = 30) have higher transcriptome variation than samples in 15 hours (n = 30). To account for the association between mean and variance in RNAseq, we first used VST to normalize the expression level for each group separately, following this document: https://rdrr.io/bioc/DESeq2/f/vignettes/DESeq2.Rmd.

After normalization with VST for each group, we find that there are difference for the overall expression variaton across groups. However, because VST estimates the mean-variance and fits the dispersal for each group separately, we are not sure whether the within-group variances are comparable among groups.

We have tried setting blind = F, fitType for all three options, and VST using one group as reference, and these methods returned similar results (while the level of difference differed). However, we feel that we need better statistical argument to make sure that the result is not the artifact of normalization.

May I ask is there any suggestion for the use of VST for multiple groups? And can we find any statistical proof (or direction of working) that this normalization method can be used for comparing variance across groups?

Best, Bitao

VST should work fine here, it estimates a simple transformation, think of it as f(x), for the entire dataset. It’s similar to log(x) but tends to have better performance than the logarithm for reasons explored in the vignette and workflow.