If you want to do away with the set screw to make the flange mounting stress free and permanent you could J-B weld it on the carb spigot if the two fit reasonably close.Don't laugh. J-B weld is completely fuel proof. The float bowl on my Iron barrel is coated with the stuff inside and out.
I would also recheck that "O" ring on the carburetor flange. It comes out really easy when mounting the carb and if it's out, just a little bit, you'll get an air leak and the engine will run poorly just like you described.
The breather hose that goes to the carb, disconnect it and just cap off the carb spigot.
Hmm, the only hose that goes into my carb is the fuel line.
Have you checked and cleaned your bike's breather hoses and catch can? If they are clogged your engine will not run good at all. Just a thought. Cheers,Bill