The victim, 31, was pushing a baby in a pram when she was approached by a man at around 4.50pm in Crow Nest Park.
Police said the man made inappropriate remarks to her before indecently touching the woman.
She managed to get away from the offender and left the park following the incident last Sunday.
The suspect is described as Asian, aged 30 to 35-years-old, and about 5ft 7ins to 5ft 8ins tall.
He had dark hair and was wearing all black clothing and a black body warmer.
PC Carrie-Ann O'Keefe of Kirklees Police, said: "Enquiries are ongoing regarding this sexual assault and we would like to speak to anyone who has information or who saw what occurred.
"We would also like to speak to anyone who saw a man fitting the suspect's description in the park shortly before or after the incident.
"Anyone who has information should contact myself, PC 4501 O'Keefe at Kirklees Police on 101 or the independent Crimestoppers charity, anonymously, on 0800 555 111."