I have data that has genetic line (line) and phenotype (pheno) as my factors and my data looks something like this:

The model I want to run is to look at both phenotype and line nested within phenotype which I believe is expressed syntactically as:

~ pheno+pheno:line

And the model matrix is:

However, I get this error when attempting to estimate dispersion:

Error in glmFit.default(sely, design, offset = seloffset, dispersion = 0.05,  : 
  Design matrix not of full rank.  The following coefficients not estimable:
 phenoL0HL:lineL0380 phenoL0LL:lineL0441 phenoL0HL:lineL0486 phenoL0LL:lineL0832 phenoL0HL:lineL0913 phenoL0LL:lineL0913

Which i think is saying that since both phenotypes are not since within a line, there is not enough representation. But a line is either one phenotype or the other.

Any help would be appreciated.

The problem here is an idiosyncracy of the model.matrix() function in R. When you try to nest one factor within another, it assumes that the levels of the nested factor start from scratch within each level of the higher factor. In your case, model.matrix() is creating extra column of 0s for each line that doesn't belong to the relevant phenotype.

Option 1. The correct way to fix this is to relabel the lines so that the numbering restarts within each phenotype:

sample_id line pheno
        1    1    LL
        3    1    LL
       14    2    LL
       15    2    LL
       25    1    HL
       26    1    HL
       27    1    HL
       37    3    LL
       38    3    LL
       39    3    LL
       49    2    HL
       50    2    HL
       51    2    HL
       61    3    HL
       62    3    HL
       63    3    HL

Then ~ pheno + pheno:line will work fine. Note that, using this formula, the LL and HL lines are treated as different even though they have the same numbering. The design matrix will have exactly 6 columns, because you have 6 lines in total.

You can see another example of this type of factor re-numbering on page 41 of the edgeR User's Guide.

Option 2. The alternative is to just use design <- model.matrix(~0+line), and then use appropriate contrasts to compare phenotypes and lines. As Aaron has pointed out, you can quite easily get any comparison from this flat model that you could have got from the nested model.

PS. Sorry, my original suggestion to remove the all-zero columns from the design matrix was too simplistic, as I should have known. It doesn't give a full rank matrix immediately and gives confusing columns, as you discovered and Aaron has explained.

This model is impossible to fit uniquely as line is nested within pheno. Any differences between phenotypes will be absorbed into the line terms, so there is no unique solution for the model coefficients. You can only do:

~line

... or if you want to account for the replicate, then:

~rep + line

To determine the phenotype effect, you can then compare the average of HL lines with the average of the LL lines, see examples in Section 3.2.3 and 3.2.4 in the edgeR user's guide.