Why did the three men leave the first two inns?

From the text,

Why, you are coming away from them. You must turn right round and go back, and then you will come to the Stag.” We said: “Oh, we had been there, and didn’t like it—no honeysuckle over it.” “Well, then,” he said, “there’s the Manor House, just opposite. Have you tried that?” Harris replied that we did not want to go there—didn’t like the looks of a man who was stopping there—Harris did not like the colour of his hair, didn’t like his boots, either.