If you're seeing any artifacts in the original video, you probably need to re-record in a higher bitrate. It needs to look identical to uncompressed. Your later encodes will be trying to encode all the artifacts in the original video, which could be why the file sizes keep getting bigger - you're giving it noisier video than the original.
50mbps for recording as an intermediate like that is well within the realm of normal. You can try having obs record in 264 with a quality setting instead of a bitrate setting, which can save space when things are more static - something like cq 6 or lower can do pretty well.
Unfortunately, yeah finding the sweet spot does take forever. One thing I would recommend is once you have an idea where you want to land, try a few much longer videos and see what the differences are. Slower paced sections might compress much better than the fast action stuff in one codec or another. Again it's all kind of a balancing act on where you want to be.
From a technical level you should use something lossless in this situation, but it really quickly becomes impractical. Actually lossless 1080p60 is going to be something like 500mbps, so if you're playing for an hour I hope you have a spare 2tb drive laying around. The artifacts in really high bitrate compressed video are so minimal that they basically don't matter. Often codecs do noise removal first thing so whatever minor artifacts still exist will get smeared over by that anyway.
Also when you are testing make sure there's some movement in the video. AV1 especially has modes for presentations and things that basically make a PowerPoint, so sizes might be unrealistic if you're just recording your desktop. I don't think that gets enabled in handbrake but it's been a while since I looked.