I believe you have been missing this (in the package vignette, section AnnotationTrack, page 33:
Unless we tell the Gviz package how to deal with the respective feature types they will all be treated in a similar fashion, i.e., they will be plotted using the default color as defined by the fill display paramter. To define colors for individual feature types we simply have to add them as additional display parameters, where the parameter name matches the feature type and its value is supposed to be a valid R color qualifier. Of course this implies that we can only use feature names that are not already occupied by other display parameters defined in the package.
plotTracks(aTrack.stacked, groupAnnotation = "group", + foo = "darkred", bar = "darkgreen")
And from the class' documentation, (? AnnotationTrack), this:
feature: Factor (or other vector that can be coerced into one), giving
the feature types for the individual track items. When
plotting the track to the device, if a display parameter with
the same name as the value of 'feature' is set, this will be
used as the track item's fill color. See 'grouping' for
details. Needs to be of equal length as the provided genomic
coordinates, or of length 1.
Florian