British Prime Minister Theresa May and other senior Conservatives told former Foreign Secretary Boris Johnson to say sorry Tuesday for a newspaper column in which he wrote that burqa-wearing women looked like “letter boxes” and bank robbers.

Johnson, who quit the government last month in a dispute over Brexit, made the remarks in a Daily Telegraph article published Monday.

Johnson said he opposed banning burqas and other face-covering garments, but wrote that it was “absolutely ridiculous that people should choose to go around looking like letter boxes.”

His article drew criticism from Muslim groups and fellow politicians —.

Mohamed Sheikh, founder of the Conservative Muslim Forum, said Johnson’s article had been “totally out of order.” Middle East Minister Alistair Burt criticized Johnson for comments he said “many people would find offensive.”

Conservative Party chairman Brandon Lewis said in a tweet that he had asked Johnson to apologize. May said she agreed with Lewis.

“I do think that we all have to be very careful about the language and terms we use. And some of the terms Boris used describing people’s appearance obviously have offended,” she said.

There was no immediate sign that Johnson planned to back down.

Latin-spouting, tousle-headed Johnson is a former mayor of London and one of Britain’s best-known politicians. He resigned as foreign secretary in July, accusing May of killing “the Brexit dream” with her plan to seek close economic ties with the European Union after the U.K. leaves the bloc next year.

The resignation solidified Johnson’s position as a leader of the pro-Brexit wing of the Conservative Party, which is deeply divided over its attitude to the EU.

Many expect May to face a leadership challenge if faltering Brexit negotiations don’t improve — and Johnson is likely to be a contender to replace her. Some suspected Johnson’s burqa comments were intended to boost his appeal among right-wing members of the party.

Sayeeda Warsi, a Conservative member of the House of Lords, said Johnson was using Muslim women as a “convenient political football to try and increase his poll ratings.”

“These were offensive comments but clever politics,” she said. “Boris knew the effect and the impact that this kind of dog-whistle politics would have.”

Several European countries, including France, Belgium and Denmark, have banned face-covering veils in public, but none of Britain’s main political parties supports such a restriction.

“I believe women should be able to choose how they dress,” said May.

(AP)