I have a bulk-RNA experiment, and I have mapped the sequencing to the human genome using STAR and featureCounts, producing a count-matrix of ENSGs (rows) and samples (columns) of size [ExS].
I'm interested in running an edgeR>limma differential gene expression analysis only on protein-coding genes. As multiple ENSGs can be mapped to a single gene (e.g., PINX1 is mapped to ENSG00000254093 and ENSG00000258724), I'm wondering about the right timing of the mapping from ENSGs to gene names, and the right time to filter out non protein-coding genes.

The way I'm currently doing this, is

1. Use filterByExpr() on the [ExS] count-matrix, making it size of [FxS].
2. running voomLmFit() and eBayes()/treat() on the [FxS] count-matrix, producing a MArrayLM object with F rows.
3. Using biomaRt, filtering the MArrayLM to include only protein-coding genes, shrinking the from F rows to P rows.
4. Using biomaRt, converting the ENSG IDs to gene names; The MArrayLM object still has P rows, but their names are now different: gene symbol names instead of ENSGs.
5. Continue on decideTests() on the MArrayLM object with P rows.

This is what I have interpreted from Aaron Lun's comment:

I would only filter out uninteresting genes after eBayes. Then you can have your cake (reliable variance estimates) and eat it too (fewer tests during multiplicity correction). Note that you should have already filtered out low-abundance genes, as these won't provide much information for variance estimation anyway.

So, I'm wondering

1. Is my approach the right one?
2. Should the steps "use biomaRt to map the ENSGs to gene names" and "filter by protein-coding" be done separately? together? on different stages of the pipeline?
3. So far, I haven't encountered any duplicated gene symbols after filtering... what should I do if they'll pop up on future analyses?

Thank you!

You say you have a matrix of counts, but that's not what you should be using. Instead you should be using a DGEList object. I would normally do something like this

> library(AnnotationHub)
> hub <- AnnotationHub()
> query(hub, c("homo sapiens","ensdb"))
## now identify the EnsDb that matches with the Ensembl build you aligned against
## I'll just assume it's build 113
> ensdb <- hub[["AH119325"]]
## where XXXX is the correct number for your Ensembl build
> annot <- select(ensdb, row.names(<your matrix object's name>, c("GENENAME", "GENEBIOTYPE"), "GENEID")
## I don't have your row.names, so as an example
> annot <- select(ensdb, head(keys(ensdb)), c("GENENAME","GENEBIOTYPE"), "GENEID")
> annot
           GENEID GENENAME
1 ENSG00000000003   TSPAN6
2 ENSG00000000005     TNMD
3 ENSG00000000419     DPM1
4 ENSG00000000457    SCYL3
5 ENSG00000000460    FIRRM
6 ENSG00000000938      FGR
     GENEBIOTYPE
1 protein_coding
2 protein_coding
3 protein_coding
4 protein_coding
5 protein_coding
6 protein_coding
## this should always be TRUE, but just to be sure...
> dglst <- if(isTRUE(all.equal(annot$GENEID, row.names(<your matrix name>)))) DGList(<your matrix name>, genes = annot)
> dglst <- calcNormFactors(dglst)
> keep <- filterByExpr(dglst, design)
> dglst <- dglst[keep,]
> fit <- eBayes(contrasts.fit(voomLmFit(dglst, design), contrast))
## now if you only want protein coding
> fit.filt <- fit[fit$genes$GENEBIOTYPE %in% "protein_coding",]

I don't ever worry about duplicate gene symbols, because I just think of them as being easy names for my biologist friends to remember. There's no guarantee that they are unique anyway. If you do have duplicates, you can always go to ensembl.org to figure out what's going on, but that is a one-off type deal, and not something I would try to do programmatically.