Kirklees Tories announced they would tear up the council's contract to take in a percentage of asylum seekers when it expires next year – and would try to end it early.
But the council's cabinet member for housing said the policy was based on Tory fears they could lose their seats in Thornhill and Mirfield to the BNP.
Coun Andrew Cooper (Green, Newsome) said: "There is a real sense of desperation when politicians try to get votes by aiming for the easy targets, in this case asylum seekers."
Thornhill Conservative councillor Imtiaz Ameen said he was confident of beating the BNP again, and said they had based their policy on what they believed in, not what would happen at the ballot box.
He said Dewsbury people could not understand how houses were found for asylum seekers at the drop of a hat, but not for them, and this was causing resentment and unnecessary friction.
He added: "There have also been a number of cases of these people begging on doorsteps in my ward, and entering houses without permission.
"Neither the police or Kirklees seem prepared to stop these individuals, and the only solution is to radically review the situation."
SYMPATHISED
Tory leader Coun Robert Light (Birstall and Birkenshaw) said he sympathised with genuine asylum seekers and refugees, but added: "The bogus refugees have caused problems throughout Kirklees, particularly in parts of Huddersfield and Dewsbury, and we intend to deal with this to maintain the good community relations that currently exist in our communities."
Sayeeda Hussain-Warsi, the Conservatives' newly-selected prospective parliamentary candidate for Dewsbury, who has worked with asylum seekers through her job as a solicitor, said she had not been part of bringing about the policy and had not yet discussed it with Coun Light.
(From page 1)
She said the Government's policy on dispersing asylum seekers was wrong and had caused friction in communities and put pressure on health and education resources, which were already overstretched.
Coun Cooper said asylum seekers would be sent to Kirklees by the Government, and put in private rented housing, whoever controlled the council.
"A very small number of people who are rehoused in Kirklees Council properties are former asylum seekers; the Conservatives know this, they have seen the figures, but they choose to ignore them for political purposes."
Kirklees Council leader, Coun Kath Pinnock, said she was sad and disappointed asylum seekers and refugees were being attacked in this way, and said the Conservatives had never raised these issues with her or in an official council arena.
She said refugee access to housing was exactly the same as anyone else and applications were processed in the same way.
Labour group leader Coun Mehboob Khan (Birkby) said the policy was an attack on groups who could not defend themselves and it undermined established migrant communities who were now second or third generation.

A POLICE spokeswoman said earlier this year three women were arrested for begging in Dewsbury but were cautioned and released without charge.