When I design fins or modify an original fin design, I just iterate until I get the stability I need for the biggest engine I'll use. First get the basic fin shape you want. If the stability is too low, increase the root chord and semi-span dimensions proportionally until you get the stability required. If too high, do the reverse. When you get it to where you want it, then adjust the TTW fin tab parameters to fit your MMT. The fin tabs slightly shift the CG back, so a final iteration may be required.Neither, I'm using open Rocket. Looks much better and now I see the problem you're pointing out with a long motor case.
Since you're light years ahead of me with rocksim, do you think cropping the fins would have any stability issues?