Dan
Flight simulation programs do not provide exact solutions of a rocket flight for a number of reasons. In the case of RS, OR and RASAero, two of the principal reasons are: 1.) simulation of a rocket flight needs all environmental factors and all the possible physics which is not realistic for a hobby simulator because 2.) the description of the rocket is not exact.
Virtually all simulators use approximate solutions to some extent which saves a lot of rocket description, measurements and computer power, but how and what approximations are made can make a difference in the simulation for "unusual" designs. If your rocket is a plain vanilla 3FNC or 4 FNC design, most sims will give very close results, but differences will arise when you deviate from the "typical" rocket design.
Sims address aerodynamic flows and shocks over aerodynamic surfaces by approximations because a detailed numerical solution would take too long for the hobbyist, and would require a more detailed description of the rocket than a hobbyist could supply. And not all sims use the same approximations. OR said your design may give incorrect results (and indeed it warned you of that) and RS apparently makes no claims. If you are capable of writing out the equations to describe the flow in greater detail, then you can get a more accurate solution, but is it really worth the effort? Most would say not and simply modify their design to obtain a stable solution. Others won't.
I started designing rockets before hobby sims existed. I first designed by "what looks right for a rocket" followed by a swing test, and this method never failed. In later years after professional training, I developed the habit of correlating cause and effect with very simple models. For fins it was quite straight forward: you need a minimum of 3 and not more than 4. After looking at a number of successful rocket designs, the FS (fin span) is usually close to 5/N times the airframe OD where N is 3 or 4: so FS ~ 5/3 the OD for 3 fins and 5/4 the OD for 4 fins with FS rarely < 1.
Is this a hard and fast rule? Absolutely not! There are a number of rockets with a FS = 1.5 with 3 fins.
Why do I use this correlation? Because I don't do oddrocs and there's nothing good about a failed flight.
Will it produce a stable rocket? Absolutely unless you mess up the CG/CP rules of thumb.
Can you reduce the FS further? Almost anything is possible, but you need to do some homework beyond the sims.
Also to your address your comments. When I simmed your rocket in OR, I loaded in the biggest motor that fit your MMT. Why? Because someday, you or another person might do it. So while my FS recommendation may be conservative, it will be ok.
Bob