When I am applying these two different designs in edgeR GLM test it gives different results:

design<-model.matrix(~0+treatment:group+group:plant,data=y$samples)
design <- model.matrix(~0+group+group:plant+treatment:group,data=y$samples)

even the groups are same like:

treatmentD:groupR vs treatmentD:groupS

Can anyone help me why it happens?

Whenever you define a model, you are implying that the things on the right hand of the equation explain something about the observed values on the left hand side. Or, put another way, you are saying that you think that changes in the expression of a gene can be explained by inherent differences between various known attributes of the things that you measured (e.g., what group it was, what the treatment was, etc).

In the first model you describe, you are saying that group membership doesn't mean anything, or alternatively, that you assume that the gene expression for all the groups should be roughly the same, and the only differences are due to interactions between the treatment and group as well as the group and plant type. In the second model you are saying you expect that the relative expression between groups may change, plus there might be the interactions we already mentioned.

When you fit the model, the goal is to partition the observed variability into the different 'buckets' you have defined in your model, in some optimal way. In other words, in the second model, you try to estimate the overall gene expression that is due to being in one group or another. In the first model you don't do that at all, and instead any group-specific variability gets subsumed into the two interaction terms, or into the error term.

Which is a long winded way of saying that specifying a different model will result in different results, because you know, you specified a different model.

Thank you very much.

I have one another question. Like, when I am using the 2nd design then I am getting same genes with opposite expression in different treatment.

For example, I have two groups R and S as well as two treatments M and V. when I am checking for groupR:treatmentM vs groupS:treatmentM and groupR:treatmentV vs groupS:treatmentV then I am getting the same DE genes with opposite expression.

-1  258
0  41589
1   119

and

-1  119
0  41589
1   258

Can anyone tell me why it is like this?

Actually, I want to compare both between and within subjects. I am following section 3.5 of this edgeRUsersGuide and I am using

design <- model.matrix(~0+group+group:plant+treatment:group,data=y$samples)

 But I am getting the same DE genes with opposite expression for groupR:treatmentM and groupS:treatmentM.

Please help me where I am wrong?