3g for a ~5" diameter tube 24" long sounds like a lot of BP, however, that doesn't sound like it is your primary issue.
One thing you did not provide - number and size of the shear pins. It sounds like your shear pins are holding on for quite a while before they shear or there wouldn't be so much pressure on your rivets. For a cardboard rocket with a (presumably) plastic nosecone, you don't need a lot of holding power for the shear pins. You want just enough holding power to prevent the nosecone from ejecting during the drogue event, but also a weak enough connection that you don't need a huge event to eject the main. In some of my rockets with a plastic nosecone, I have even been able to get by with a friction fit.
Personally, I would not use pop rivets. They can work fine, but using PEM nuts in your coupler and screws to hold the payload bay on offers some advantages. First, it is a heck of a lot easier to take out/install a screw for each flight than it is to cut off a pop-rivet and then install a new rivet each time. Second, you can tighten a screw a lot more than a rivet, which helps to prevent tear out. Rivets can make a somewhat tight connection, but there will always be a little bit of "play" - a loose connection will tear out more than a tight one. Third, the surface area of the PEM is a quite a bit more than a rivet and the PEM is epoxied to the coupler, so it has less of a probability of tearing the tube apart.
