Certificate Encryption

I have a secured PDF with Certificate encryption.  If I have to add a new user's security key to grant them access to the PDF it seems like I lose the current users security access to the PDF.  Is there a way around this or do I have to add all users every time there is a additional user to be added.

Thanks

Al