I general, I simply don't let anyone hold my passport at any time for any reason. I am polite but firm and usually the other side will eventually back down.

The sole exceptions:

If you are arrested, the police have the right to hold your documents. This is true anywhere. Just try to not get arrested...

When you get a new Residence Permit, your school must send your passport to the police. The total period out of your kung fu grip should be no more than 2 weeks. If it doesn't come back as agreed without a reasonable explanation, I tend to get ugly.

Passports are one place to take things seriously...