I was cutting a round hole in 3mm mdf material with a 3mm round nose end mill on the inside of the line using very low speeds and feeds and found that the accuracy of the diameter was a long way out
Was trying for a 40mm diameter and it finished up approx 41 mm
Redid it and reduced the dia to 39mm and still bigger
Redid it and allowed an offset but still bigger
Am using woodworker X50 original Vcarve Pro v 12.505
How long is the endmill?
And did you check its runout?
Are these pocket cuts or profile cuts?
Did you try a flat endmill?
I get accuracy on my machine within 0 0003”
Also consider the accurate measure of the endmills you are using and perhaps calibrating your axes’ travel. Sometimes suboptimal feeds and speeds can lead to rubbing vs cutting which can lead to inaccurate machining. I work mostly in aluminum and for tight tolerance parts will always include a final at depth ‘spring pass’, which may help. Perhaps check your mechanicals to minimize backlash if present.
I personally use a 1/8 up cut bit at 22,000 rpm with a feed of 100 inches per min cutting full depth
I would use a profile cut inside the vector line run a cut at 39mm then at cut at 40mm with a follow up spring cut. I machine a lot of kiln dried oak and usually achieve very similar to what Ken5 has posted
When you say motor resolution, did you use the Wizard in Masso or how did you set this up? I havent gone this far. I did however measure and setup the backlash adjustment which has been really helpful as it corrected about 5 thou of backlash now.