You can stop the slop (slightly bigger hole) if you drill a hole in a piece of 16mm or thicker mdf in the drill press and use that as a drilling jig by clamping it where you want the hole. The MDF jig helps keep it all vertical and stops any initial wander. Best to put a large x on the blank so you have lines to line up and ensure it is centred.
I had to use this tecnique with a hole saw that was too deep for the drill in my arbour. Worked well.
I never make mistakes, I thought I did once but I was mistaken
Top 10 reasons I procrastinate