Holgrave is described as behaving "as if the garden was his own." What might this allude to?

Holgrave is becoming extremely comfortable in the home. He is presumptuous, and his presumptions succeed in discomfiting Phoebe, who finds his actions and speech regarding the garden authoritative and parasitical. This might possibly allude to some sinister intent on Holgrave's part. None-the-less, Phoebe shows the man a kind of deference in her ensuing actions;

Silently, and rather surprised at her own compliance, Phoebe accordingly betook herself to weeding a flower-bed, but busied herself still more with cogitations respecting this young man, with whom she so unexpectedly found herself on terms approaching to familiarity. She did not altogether like him.