This is an issue that has been debated here before. I believe it is correct that drift is proportion to descent rate and wind, period.
Based on empirical evidence, and some simple math, most folks would agree with you. Anything that slows the descent rate necessarily increases descent time, and therefore drift regardless of the form factor. Stability, and drag, are functions of design and execution.
I find x form parachutes over rated, as they attempt to provide the stability of a larger parachute with spill hole in a different form factor.......and usually fall short of expectation, as they're especially prone to spiraling themselves up into streamers without a 'good' ball bearing swivel. Essentially an answer in search of a question that's better answered with a smaller parachute, or the same size parachute with a larger spill hole.
Yes, x form reduce drift over compared to the same size flat sheet without spill hole. BUT, in doing so, they INCREASE descent rate. If you're already at the high end of acceptable descent rate, a same size x form will only make things worse. If you go UP in x form size, you're back where you stared in descent rate, BUT you now have all the x form 'issues' inherent in the design.....and perhaps added bulk that you don't have physical room and/or newtons for.
Yes, x form may reduce swaying compared to the same size flat sheet without spill hole. BUT, in doing so, they are prone to rotating, and without a swivel are prone to rolling up the shrouds and acting like a streamer. They are also prone to causing the whole rocket to rotate, so if your fins are prone to damage 'swaying', they'll be put MORE at risk of damage twirling.
The answer to 'Should I use an x form?' is almost always answered with 'No, tune the parachute that gives you acceptable descent rate with spill hole size FIRST to prevent swaying. If you find you need more parachute to slow things down, go up one size, and continue tuning with spill hole size."