You're right but you don't want to be limited by a smaller diameter stack.
Velocity has to increase to maintain the same flow rate in a smaller stack. Let's say you want a flow rate of 2 cu ft/min to maintain temp and you have a 4" stack. The air velocity thru the stack is 4.5 in/sec. Make that a 3" stack and the velocity has to jump to 8.5 in/sec to maintain the same flow rate. The same amount of air is moving thru the smoker but the speed of the air moving thru the stack has changed. With a 4" stack, a damper can effectively choke it down if desired, giving you another level of control. What you don't want to happen is adjusting your intakes to control temps only to find your stack is limiting you. Friction plays a role here but the numbers don't take that into account. Since a smoker is basically a natural convection system any obstacle in the airflow affects it.
Using RC's favorite analogy of having the smoker upside down and pouring water in the FB, too small of a stack limits how much water can move thru the smoker.
The part of the equation I'm guessing at is what IS the optimal flow rate thru a RF smoker? I used 2 cu ft/min but it's probably more. Where's Frank?