Sadly my knowledge has faded with time. I could dig out my textbooks and refresh my memory but that would take time.
In a greatly simplified form, a system with negative feedback will be unstable (oscillate) if the phase shift on the output is 180 degrees. Add this to the 180 degree phase shift inherent in negative feedback and you now have positive feedback. If the loop gain is more than one, the system will oscillate.
Note that you want the gain to be high because the steady state error is dependent on the gain. Higher gain means less error.
Another
report I ran across was for a control system designed for the Paiute-Tomahawk sounding rocket and used a compensator which had two poles. Alas, the reference in the paper was to "some dude" so they didn't show their work. That one was particularly relevant because it used small forward fins to control pitch and yaw.
Other than that annoying missing detail it is a quite detailed report showing the hardware design and even includes the program listing.
As for non-linear dynamics, almost all engineering analysis tools assume/require linear systems. Non-linearilty makes life very hard and while there are ways to cope (usually involving linearizing around an operating point), there are still dragons lurking.
While my skills are from the Electrical Engineering side, control systems have a lot of overlap. So here is a stab at the roll control problem.
First up I am going to assume that absolute roll orientation isn't required so that what is being controlled is roll rate. This is important because this improves phase margin by 90 degrees. (If you control roll you already have 180 degrees of phase shift baked in.)
There are of course other sources of phase shift in the control loop.
The sensor has some unknown phase shift due to its sampling and filtering. One of my big gripes with the MEMs sensors is while they include digital filters, they don't tell you anything about their performance. Which is really important here because all filters alter the phase. Some more than others.
The sensor data then goes into the micro-controller which updates periodically. The period matters because that represents a delay which is a frequency dependent phase shift.
In this case the controller then outputs a periodic pulse to control the servo. I have noticed that digital servos are being used which is good because some support higher update rates. Which means less delay and phase shift.
The servo of course has its own feedback system to control its output position. This needs to be included in any model because once again, it adds delay.
Finally the fin will create a force which torques the rocket. The rocket has inertia which will resist any change of course. A lower moment of inertia means that the loop has higher gain so a smaller rocket could oscillate if you use the same gain as you used with a larger one. I pulled up a couple of Rocksim files for some 4 inch class high power rockets and after motor burnout the moment of inertia in the longitudinal axis was on the order of 100 times greater than it was in the roll axis. So the same thing applies there. A gain that is fine for pitch/yaw could result in unstable roll control.
So just from basic principles I can see that it is likely that there will be some frequency where the phase shift is 180 degrees. Which means that the loop gain must not be greater than 1 at that frequency. Which means that the gain must drop with increasing frequency. Which means including a low pass filter.
Alas, low pass filters add phase shift as well so the pole (corner frequency) must be well placed.
Another issue to be dealt with is the update rate on the servos. Standard is 50Hz and that causes trouble. The control system must have more bandwidth than the system being controlled. A Saturn V would probably be fine with a 1Hz rate because nothing happens fast with something that big. But as things get smaller the frequencies get higher. You would like for the maximum frequencies in the system to be no more than 1/10th of the update rate to keep Nyquist and friends happy.