Edited the original question to make it a little clearer:

I have a gene expression data set of 165 samples and different subject/ Time-point at which the sample was collected ( after a certain dose of drug and days), source of sample, whether it was CD33/34 enriched or not. 

A subset of this actual data looks like this:

I concatenated time point, sample source and sample type as the grouping factor:

design <- model.matrix(~0+TimePoint+Source+Sample.Type)

My question is that not all subjects have the same matching time point collected or even same sample source. For example, I wanted to look at the differences of CD33/34+ PBMC samples between samples collected after Dose1Day8 and after Dose1Day1. There are 20 CD33/34+ PBMC samples from Dose1Day1, only 7 of those from Dose1Day8.

Furthermore, only 5 subjects are present in both time points.

Do I only look at those 5 subjects (making the comparison balanced and 10 samples total)? If that's the case then I'll have to filter the samples first instead of fitting all the data to one glm model?

Or, do I look at all the available samples within that group and make the contrast looks something like : 

Group = factor(sampleInfoAll$Group)
design <- model.matrix(~0+Group)
my.contrasts = makeContrasts(Dose1Day8.PBMC.CD33_34pos_vs_Dose1Day1.PBMC.CD33_34pos=Dose1Day8.PBMC.CD33_34pos-Dose1Day1.PBMC.CD33_34pos,levels=design)

You mention that you have multiple samples from the same subject. However, your design matrix does not contain a term for Subject. This will violate the assumption of independence for the random component of the GLM. Based on the subset of samples in your post, I see two approaches to restoring independence:

1. Keep only the subjects with samples from both time points, and add Subject to the design matrix. (Technically there is no need to remove the other samples, but they won't be of much use in the GLM anyway as they will always be fitted perfectly by their own subject-specific term.)
2. Keep only one sample from each subject (i.e., discard one from each pair of samples from one subject).

The idea here is to avoid dependencies between samples from the same subject, either by modelling them explicitly in the model in approach 1, or by removing some of the relevant samples as in approach 2. Given you have 165 samples, I would recommend approach 2 as it retains most of the information in the data set (only 5 samples lost).

I would also suggest using the Group term in your design matrix rather than adding Source, Sample.Type and Timepoint, as the former parametrization avoids making assumptions about the additivity of the latter effects.

Finally, you mention two time points, but I can see at least one more: Dose1Day2. I'll just ignore this.

If you use edgeR to do the analysis, you must include Subject in the linear model -- see Aaron's answer.

Personally, I would use limma to analyse an experiment like this and would use duplicateCorrelation() to model the correlation between repeat observations from the same subject. limma is able to compare treatments even when they are not applied to the same subject, in other words it can combine both within subject and between subject comparisons.

The edgeR approach is more conservative. The limma approach recovers a lot of extra information from the between-subject comparisons. It depends on how conservative you want to be.