The protests followed the arrest and charge of an asylum seeker on suspicion of alleged sexual assaults in the town.
Whitbread repeated calls for the hotel to stop housing asylum seekers following the incident.
Six people have been arrested following the latest protest with one man accused of violent disorder.
Whitbread said there were concerns about the location which is near a school and the town centre.
“Fair-minded people within the local community, initially protesting peacefully, want the hotel closed,” he said.
“I have to say, as the leader of the council, that’s always been my position and the council’s position. This is just not a suitable location. The sooner it’s dealt with, the better.”
He said he sympathised with the “vulnerable” asylum seekers at the hotel but stressed locations which had “less impact on established communities” needed to be found by the Home Office.
“It’s a powder keg now and we need to get something done and we need the Home Office to listen,” he added.
Asked if he had a message for Mr Yaxley-Lennon and his supporters, Whitbread said: “Keep away, [you’re] not welcome here in Epping Forest.
“What we want to do is make sure our town stays peaceful and people can enjoy their lives here. That’s the most important message to get through.”
Mr Yaxley-Lennon has been contacted for comment.
