Ideally the bigger the better for the vent cross section.
However if you vent directly outside through a large vent then considerable noise may escape as well.
To counteract that some sort of "S" or "double S" muffler/baffle box is needed to reduce noise transfer - if the path is is too tortuous or too small it will generate back pressure and reduce flow.
So now were talking a bigger vent to cope with the baffle box.
I'd say it needs to be at least as large as the cross section of your main ducting trunk line but if you can get it 2x that cross section it will be better.
I use 4x the cross sectional area of my 6" trunk line - the net effect is about a 1% loss in pressure. Halving the cross section will increase the back pressure to more than 2%.
A technical post about all this is available here
https://www.woodworkforums.com/f271/.../2#post2127617