Noise is not much of a problem, because you integrate your accelerometer data. The primary concern is nonlinearity and maybe things like offset drift.Did some quick math, here are the expected accuracies, drifts, etc

For the IMU I chose, the ICM-20602, the noise is 1mg(RMS). Therefore the velocity drift is expected to be at 0.0098 m/s/s. This is with a max readable acceleration of 16g

The only IMU I could easily find with more than a 16g limitation, the LSM6DSO32TR(with double the max readable acceleration, at 32g), has a 5.2mg(RMS) noise floor. The velocity drift is 0.0529 m/s/s.

Here's a table with the expected tolerances, accelerometer alone

Seconds since arming ICM-20602 Velocity Error Apogee Error LSDM6 Velocity Error Apogee Error 60 s 0.588 m/s 17.64 m 3.174 m/s 95.22 m 120 s 1.176 m/s 70.56 m 6.348 m/s 380.88 m 300 s 2.94 m/s 441 m 15.87 m/s 2380.5 m 600 s 5.88 m/s 1764 m 31.74 m/s 9522 m

I'm really tempted to go without a barometer if the ICM-20602, at least for the small flights I'm most likely to go for that's a perfectly fine velocity/apogee error for me.

Your table bases the error on "Seconds since arming". You can compensate your sensor offset right until liftoff, or at least keep the altitude and velocity values fixed to zero before liftoff is detected. This will reduce your errors significantly.

Reinhard