“Severe action” is needed after Boris Johnson said women in burkas look like “bank robbers”, the founder of the Conservative Muslim Forum has said.
Tory peer Lord Sheikh has called for the whip to be taken from the former foreign secretary.
He told the BBC: “Take the whip from him. Why not? He’s not a super human being, he’s a member of the party.
“The party chairman, the prime minister has the right to take the whip… that’s the thing I’d like to see.”
His comments come as Theresa May backs calls for Mr Johnson to apologise.
But he has so far defied mounting calls for him to retract his comments.
The prime minister said the remarks “clearly caused offence” as she told him to say sorry.
Mrs May said: “Some of the terms Boris used describing people’s appearance obviously have offended.
“What’s important is do we believe people should have the right to practise their religion and, in the case of women and the burka and niqab, to choose how they dress.
“I believe women should be able to choose how they dress.”
A source close to the former foreign secretary told Sky News it was “ridiculous” his views were being “attacked”.
They said Britain must not “fall into the trap of shutting down the debate on serious issues”.
The source added: “We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists.”
Former UKIP leader Nigel Farage urged Mr Johnson to stick to his guns, telling Sky News he should “stand firm” and not apologise.
Mr Farage told Sky News: “There’s a big question here. Should public figures be allowed to say what they really think without being shouted down and the constant demands for them to apologise?
“I don’t know what Boris will do. In the past we have seen him back down and apologise for things he’s said.
“My view is on this he really should stand firm and he will have a big majority of ordinary people thinking he’s got every right to say this.”
The controversy began when Mr Johnson wrote a piece for the Daily Telegraph on Monday, in which he argued against following Denmark’s lead in banning the burka.
But he made several comparisons between the covering worn by some women, saying it made them look like a “bank robber” or a “letter box”.
Mr Johnson also revealed he would “feel entitled” to ask a woman wearing the burka to remove it if she came to his constituency office.
He concluded: “In Britain today there is only a tiny, tiny minority of women who wear these odd bits of headgear.
“One day, I am sure, they will go.”
The comment sparked a backlash from senior Conservatives, including Brandon Lewis, the party’s chairman.
He revealed on Tuesday he had asked Mr Johnson to apologise.
Baroness Warsi, the first Muslim cabinet minister and a former Tory party vice chair, also told Sky News that Mr Johnson was using the issue as a “political football” to “increase his poll ratings”.
She said: “Mr Johnson is making yet another leadership bid, and he will say and do whatever needs to be said and done to make that as successful as he can.
“I hope he will be disciplined but I’m not holding my breath.
“Sadly this is not the first time that such comments have been made within my party.”
Baroness Warsi added that when Conservative politicians made “crass” comments about British Muslims, it “doesn’t diminish their political status, it actually increases their political success”.
Conservative MP Nadine Dories responded on Twitter: “Any clothing a woman is forced to wear which hides both her beauty and her bruises should be banned.”
She added: “Boris didn’t go far enough.”