I notice that some McMaster-Carr mounting hardware comes one piece at a time, but they're probably cheaper at Digikey.
There's an outfit called Micro-Fasteners, though I haven't bought anything from them. I think they cater mostly to the RC hobby market, or at least they have advertised in those places. Good for screws, washers, etc.
M3 threads are just a little larger than #4, M2.5 are about .014" smaller. Maybe a #4 screw can almost fit in a hole intended for a 2.5mm screw with generous clearance. #2 and M2 are annoyingly close. An M2 screw can almost work with a #2 nut until vibration starts. M5 and #10 are annoyingly close, too. I don't remember if it's 10-24 or 10-32 that almost fits.
I don't know where you can buy small numbers of them, but if you don't like using thread lockers on everything, you might try square cone washers or belleville washers, which can keep things tight even if there's a bit of expansion or contraction. You can get assemblies called SEMS, where a lockwasher, square cone washer, etc. stays retained on the screw. You can also get nylon insert locknuts, which are great as long as you don't use them too many times. You can also get screws with "patches" or coatings that will make them more resistant to vibration. The patches are little bits of plastic set in holes that are a bit more snug than the rest of the thread. One coating is called Vibra-Tite. We used to use that stuff at one place I worked, and at other places I've seen hardware that had coatings which looked similar. Again, I imagine that these work best the first few times and then degrade. Unlike thread lockers, something like Vibra-Tite doesn't make it hard to unscrew the fasteners.
A lot of the fasteners I use come from a grab bag or assortment I got from one of the surplus places. A bit wasteful, as I will probably only ever use a couple of percent by weight. Probably All Electronics, but they are defunct. Other surplus places are Aretronics (where some people from All Electronics went), Electronic Goldmine, and American Science and Surplus.