I've had the reverse of this situation happen to me when an expat (married to a Chinese woman
and who have had a child together), chooses to speak to me using just Chinese. Is it a
test I ask myself? You don't see this happening with Chinese teachers of English when
they get together.

Chinese teachers in China are a poor comparison to your experience.

Imagine a Chinese national in Miami who is teaching Chinese in U. of M. meeting another Chinese citizen who is also living and working in the United States, perhaps also married to a U.S. citizen. Would you find it surprising if they spoke English to each other when meeting on the street?

Or French in Paris?

It may very will be that he is 'challenging' your Chinese ability. But it is interesting how powerful, and perhaps presumptuous, our expectations for English usage are among native English language users.