He broke the contract first. I just got tired of all this stuff and walked away. Then he contacted my college that holds my visa and foreign experts certifcate and threaten the college.

In that case, your work at the school is not legal, it seems to be either the case that he can't do shit because you wasn't working legally for him in the first place, or you should have never revealed your details to him, so he can't grass you up to your college. Anyway, although you can get in shit for doing illegal part time work, if he grasses you up, he gets himself in shit, so I doubt he'll do anything really.

If the college was farming you out, then it is maybe also the case that your college was breaking the law. Especially if that place had no licence to hire foreigners.

I could be wrong, but that's my impression.