In terms of limma/edgeR usage, this seems to be a weird question. However, my question is for such situation: Assuming I collected lots of single cell datasets, and I have a set of candidate genes. What I want to ask/test is that: Is there some genes only expressed in some cell types ?

I can build linear model use limma and test whether some genes differential expressed between different types of cells.

fit = lmFit(y, design = model.matrix(~ 0 + celltype + other_covarites))

But here I change the null hypothesis to a gene is not expressed in all cell types, it seems that I need a hypothesis test to test on mean expression above some threshold but not logFC. I am not sure I asked the right question and is that possible to do that ?

Any advice would be appreciated.

There is no formal way to test "expressed" vs "not expressed", partly because there isn't a clear definition of what "not expressed" means. Nevertheless, one practical way to address the question would be to look for genes that are

1. Differentially expressed between groups and
2. Have average RPKM value less than some threshold one of the groups.

For example, you might look for genes that are significantly DE, and have RPKM less than 2 in one group and have RPKM more than 10 in the other group. I'm just choosing arbitrary thresholds here. You could choose other thresholds that are appropriate for your experiment.