I choose not to 'teach grammar' in China because Chinese teachers do a relatively good job teaching students English grammar and I believe, as you suggest, that grammar is not important in oral language. Chinese students are already hesitant to risk speaking because of their fear of making errors; additional pressure is counterproductive in supporting their oral language practice.

The problem with double negatives in non-native speakers is not grammatical; double negative use by Chinese speakers impacts meaning and comprehension. Misuse of double negatives obscures understanding.

If a Chinese student says to me: "I don't really not understand."... I'm lost.