I am trying to plot a sample heatmap for RNA-seq data. I used rlog to transform count data since I have only 6 samples. The sample heatmap did not have a good cluster so I tried transforming the data with VST. I then tried to plot the standard deviation against the mean of transformed counts by rlog and VST, both figures have a log scaled axis (see figures below, left is SD & mean of rlog count, and right is for VST counts) Could you please explain why the SD of the rlog counts looks that way and is this abnormal? I was expecting that after the transformation, the gene with higher mean values will have SD scattered in the same range (similar to the right figure) meaning that the data becomes more homoscedastic, is this a correct assumption? previously, I used rlog on data from another cell line (the SDvMean plot looks okay for that one), so I prefer to use rlog for consistency, when possible.

I'm guessing this data does not conform well to the assumptions of the rlog, which are that variation is distributed a certain way from expected value across all samples.

We don't recommend rlog anymore in the current vignette or workflow, because the VST appears more stable across many different types of dataset. So I would just drop rlog and use VST here.