I'd like to think I generally am moderate.
I think you've got to be a bit crackers to be racist, homophobic or whatever. And so what the OP is espousing here isn't something I could judge him for. However, I take issue at people's concerns with mass immigration being automatically branded racist. Sure, racist people get on board with that doesn't automatically mean that everybody that's concerned about immigration is racist. It's a subtle nuance that is overlooked and it becomes a sticking point when trying to find a common ground on such an emotive subject.
The OP's strategy was a common naivety that so often prevents the subject being discussed sensibly. Alienating the people you hope to educate is a fatal error. Talk to them about their opinion rather than labelling it with a nefarious motive you've simply decided upon.