Where am I going wrong?

I am making an adirondack chair out of Tas oak. I traced around the patterns and then cut out the components around 3mm outside the pattern. I then screwed the pattern to the cutout and used a flush trim router bit with ball bearing to bring it back to size. As can be seen in the pictures it didn't go all that well with some horrible tear out in places. I am guessing that where I got tear out is because I was going against the grain. I have only done this once before but I was then using Huon and Celery pines so soft and very close grained timber and it worked fine.
I might be able to save this bit by using a hypodermic to get glue into the splits. But regardless, what is the correct method so that I don't ruin the rest of it. Or should I give up on Tas oak?
Attachment 532328Attachment 532329

3mm is way too big a bite when working against the grain with most timbers, let alone any of the more splintery ones. Try taking bites of 1mm or less working your way to a final pass of no more than .5mm. I've been routing between zero and 5mm off some Spotted Gum recently and have found it to be a minimum of 6 passes to get to the finish line. The quality of your cutter can also make a difference.

You could cut out two templates and fix one each side so you then can flip the job depending on grain direction.

+1 What aldav & Rob said. Also, to save having to make holes in your work why not use double sided tape?, I've used the stuff for a number of years now without issue. The stuff I buy is called 'XFasten Double Sided Woodworking Tape' or 'XFasten Double Sided Tape' both are seriously strong and leave no residue on surfaces.

The two template idea is the best IF you want to use the router to shape them. My method is to make one template out of 6mm MDF; tape it to another piece and hit them with the router & bearing guide to make an exact copy (if I want to make it again I make at least two copies; coat the original in thinned down varnish and store it as a "mother" template). Then attach one template to the workpiece and do as much cutting as possible with the grain. You can then attach the other template to the other side feeling the edges to match them perfectly.

Also agree with double sided tape; it goes on easy and all the load is in sheer so providing you're not holding the workpiece like a gorilla the templates won't budge.

However there are other solutions; the curve looks pretty gentle to me so I'd probably hit them with a belt or disc sander to within a bee's winky of the pencil line and then hit it with a spokeshave.

Double bearing router bit is another idea. You just flip the work over to get the optimal grain direction.

Agree with mic-d on single template. However I use two router bits. One with a top bearing and another with a bottom bearing. Same same, but different.