Clough House in Leeds Road, Birstall, dates back to 1799 and is now empty and decaying.
Rosie Carr, of York, has submitted a planning application to Kirklees Council to clear the site and build new homes.
The owners previously ran a haulage business from the building, which has since been a target for vandalism and theft.
A heritage report published on the council website says the house has been subjected to a “soft strip” with internal fixtures and fittings removed. Fireplaces and staircases remain but “they are all modern.”
The building isn’t listed for its historical merit but a request has been submitted to English Heritage.
The heritage report describes the house as a “simple Georgian building with little embellishment” and also of “modest scale.”
The report adds: “Internally there is no evidence of any historical features remaining. It should be pointed out that this was the case before the strip-out commenced.
“The central staircase is simple and not original, there are no original fireplaces, panelling or plasterwork. The roof has been replaced including the majority, if not all, of the roof timbers.
“Typically, these types of properties would be associated with the textile industry which was extremely important in the area during the 1800s.
“Clough House is, however, rather less grand than the mill owners’ houses of the same period. It is small and it has little or no embellishment and lacks the general grandness associated with wealth.”
The report says the house may have been built by a landowner or a farmer.
It adds: “What can be said for certain is that whilst Clough House is an old building it is not, in terms of its architecture and historical connections, of great interest.”
Local people have expressed concern about increased traffic levels and pressure on local services including schools.
Birstall Tory councillor Mark Thompson said he had “mixed feelings” and added: “It is always concerning when a character building is pulled down but I also understand the owner of the house wanting to do what they think is right for them.
“Just because something is old doesn’t mean it’s historic and there is a lot of land there. It’s better to use land like this than build on greenfield sites.”