why is the house haunted?

I'm not really sure that the house is haunted. While it is very easy to interpret the events in the poem as menacing and even somewhat horrific, the reality is that nothing takes place which can accurately be described only as menacing and horrific. The poet imposes no conventions of genre or moral judgment upon the action; if any is found there, it was brought there by the reader. All that really exists is a celebration of the mysterious and the unexplained that exists all around us. “The Listeners” is asking us to listen to the world around us as a way of suggesting that we just discover something mysterious even in the commonplace.

The house has been imagined of being haunted because there was no answer by anyone from inside of the house to the travellers questions.Thus it's said that the house might be haunted as there is no one inside it except some phantom listener.