I want to do gene DE for my cohort (40 cases vs 20 controls) with several covariates (age, sex, subject, RNA integrity number) in edgeR. However, I get the following warning in estimateDisp step (see below). I have read the edgeRNA manual and I assume this is due to fitting a lot of models (or lack of replicates) ? If so, however, I dont know which of suggested choices (2-4 ) is the best alternative? and how to do this (for exmple option 3). Thanks

design <- model.matrix(~ subject + age + sex + RIN + treatment)

Warning message: In estimateDisp.default(y = y$counts, design = design, group = group, : No residual df: setting dispersion to NA

The error message indicates that there is no degrees of freedom for you design matrix. I guess your subjects have many different ages and you incorporated age into the design matrix as a factor with many levels. You could try using a spline curve with 3-5 knots to represent age, which would save you many degrees of freedom. This approach is described in the limma user's guide Section 9.6.2.

You have plenty of replicates but you have formed the design matrix incorrectly. It is never appropriate to create a design matrix with more than 60 columns, as you have done here.

I assume from your question that you have 60 patients in all and treatment is a factor with two levels (case and control). You actually need:

design <- model.matrix(~ age + sex + treatment)

You can't include patient in the model when every patient is different! You could add RIN, but you may get perfectly good results without that complication. Only do that if you are an expert.

As Yunshun Chen has pointed out, it is important to ensure that age and RIN are numeric covariates (and not factors). You could try Yunshun's suggestion of a spline trend with age, but I suggest you try the simple model above first. Again, the spline curve is for experts.