Like the title says, I would just like to know if there is any problems in using samples that have very different raw counts (e.g: 1M raw reads vs 20M raw reads), even if I am applying appropriate normalization.

TMM will not work well for samples where the library size is so small that most of the counts become zero. If the library sizes are sufficiently large to have reasonable coverage of expressed genes, then differences in library size do not pose any problems and have little effect on the effectiveness of normalization.

A library size of 1 million is on the small side, but is probably ok.

Visualizing the results and inspecting whether replicate samples that are expected to cluster together do, or if the samples with extreme differences appear as outliers on plots, may help. E.g., you may want to try the plotMDS command in edgeR.