can you post pics of the hopper assembly? actually we need to know the rpm of the auger shaft, size of the auger, estimated BTU capacity of the hopper system, what sort of burn pot you are using, how the airflow enters into the burnpot, etc.
I have built many pellet smokers both from my own homemade/ modded out hoppers and also from the SmokeDaddy system. I always shot for a 4" stack on cabinets but I always wound up installing a damper and closing it mostly off.
Honestly, I don't think your issue is with the stack, I think it is an issue with the auger system.
so that is one heck of a nice pellet smoker!
The RPM on the auger is a little bit high I think for that size auger assuming the pitch is the same as the most common auger flights that diameter. if the auger is the same pitch as I think it is, and the diameter is 1-1/2" then the BTU's on a 2 rpm motor is 36,000 BTU's. if you can collect the pellets as they are dispensed for exactly 10 seconds or so and weigh them you can calculate the amount of btu capacity of your system. average cooking pellets are approx. 8500 btu's per pound. I have made cookers that are 48x48x48 with insulated cook chamber (rotisserie) and used a single 36,000 btu hopper. I think one that size would be close for yours.
the cause of the burn back on the auger tho, I think is caused by the end of the auger being above and too close to the burn pot.
here's the first one I butchered together... viewtopic.php?f=108&t=1175
here's the part of the same thread where I made a ramp to get the end of the auger far enough away to prevent burn back. this cooker is still in use by the owner today... viewtopic.php?f=108&t=1175#p15816
Hope that helps!!!
Thanks Frank! As you mentioned the stack is fine and flow is great... the issue is the rate of pellets was causing too large of a fire (flame up) in the pot. I reduced the rate % and resolved the issue. It will get a small amount of smoke in the hopper on higher temps about 400f which makes since as you are going to have more flame/combustion to reach the higher temps. I also backed the auger tube out a 1/2".
Once I complete my raspberry Pi build I will have greater control over the overall output of the pellet mechanism.