Hey Greg,
do you use the hand trim router and you switch it off over night when you pause the program? The steel shaft of the router motor and the milling bit both expand with heat. A steel shaft of 200 mm length expands by 0.14 mm (144 µm)¹ with a temperature change of 60 K. And even if you do not turn off the router, even when not milling (turning idle), the temperature changes compared to milling.
See also
1. The coefficient of linear thermal expansion (CLTE) of steel α is approx. 11–13 · 10−6/K, α of aluminium is 23.1 · 10−6/K