It's an interior cut, and the workpiece wanders away from the fence? Sounds like you have a climbing cut with respect to the grain. Try running the workpiece in the other direction, so that the cutter pulls the workpiece against the fence.

Joe