The only "white" British colonies that I know of today are Canada, USA, Australia and New Zealand.

Technically, (based on racial predominance) all other colonies that the British laid claim to can be called "black"; i.e. Caribbean nations, India, Pakistan, Malaysian islands and various African countries.

Except for the Caribbean island nations (where the "natives" were decimated or exterminated), the other former colonies generally had Natives (darker complected that the British conquerors) already firmly settled, who had their own languages/dialects. English, for the main part, was an add-on (second) language for most inhabitants, which the British insisted be used as the "official" one.

Do I think this is Fair??
No. It is a narrow, bigotted, prejudiced point of view.
Does it (partly) explain the difference in terms?? Yes.