Hey Mike,
if you really loose detail after having switched from the trim router to a spindle (which mostly have less than 5 µm runout and are able to hold their speed strictly, regardless of load (what a trim router is not able to), after thinking about it for a while, I came to the thought, is it possible that you made a mistake when mounting the collet?
ER collets are not mounted the way a trim router collet is mounted. Most people put the collet into the axle of the spindle and then they put the collet nut over it. That is definetely wrong! An ER collet has a groove that goes all the way around near the top of the collet. The corresponding collet nut of an ER collet has an excentric extractor ring or offset lip inside. It is absolutely mandatory to fully engage (clip in) the collet inside the nut before you tighten it onto the holder, as shown at this moment of the video “ER Collets | 5 Things to Know for Beginners”. See also this picture from my spindle manual:
– Source: Mechatron HFS Spindle User’s Manual
ER Collets | 5 Things to Know for Beginners - Youtube
ER Collet Essentials. Do You Know? – Haas Automation Tip of the Day - Youtube