Unclear if the OP is talking about regular primer or filler/primer (a common source of confusion on the forum). In my case, it's filler/primer, and that *sounds* like what the OP is discussing.
To a certain extent this question depends on the size of the rocket. For my LPR stuff, I'm able to hold the rocket inside my garbage can while sanding, so the very vast majority of dust stays inside there. I also have a little DC fan with a HEPA filter strapped to it that I can set next to the garbage can for when I need to sand outside the can.
There will still be dust all over my hands; I keep a damp paper towel handy to wipe off frequently. I do find the dust and the task in general to be unpleasant and annoying, but also worth it: when I experimented with skipping the filler/primer step with one rocket it resulted in noticeably more surface imperfections on the finished product.
I can't imagine what it's like to sand large rockets; fortunately that is something I don't need to worry about.
(For what it's worth, I also find the dust generated from sanding CWF to be awful as well. This contributes to my preference for papering fins.)