I would say you won't have a problem if your fillets are different in size. The main stability comes from the fins and the relationship between your Center of Pressure and your Center of Gravity. You want your CG closer to the nose than your CP.
The fillets will have very little effect on this relationship.
Fillets affect drag; they add a little bit more area to the width-wise cross section of a rocket, but not so much that you're going to lose a noticeable amount of altitude.
And because epoxy is >relatively< lightweight, it won't shift the CG much.