I am filtering the data that I am working with with nsFilter, 

filtered <- nsFilter(eset_rma, require.entrez = TRUE, remove.dupEntrez = TRUE, var.func = IQR ,var.cutoff = 0.5,var.filter = TRUE,filterByQuantile = FALSE, feature.exclude = "^AFFX")

when I use filterByQuantile = FALSE, instead of TRUE, what is this doing to my data?

after filtering I am using the limma package to decide which genes are different in each condition, and when I use filterByQuantile = FALSE I get more genes, why is that?

Please don't do this filtering. Using nsFilter() with var.filter=TRUE will sabotage any DE analysis you might do using limma. It would be better to take a simpler common-sense filtering approach, or even not to filter at all.

Have a look at previous posts on this topic, for example:

  Is filtering necessary to extract significant DE genes with limma ?