what is especially different, from the rest of the town, about the house that joe has built?

From the text:

It had two stories with porches, with bannisters and such things. The rest of the town looked like servants’ quarters surrounding the “big house.” And different from everybody else in the town he put off moving in until it had been painted, in and out. And look at the way he painted it—a gloaty, sparkly white. The kind of promenading white that the houses of Bishop Whipple, W. B. Jackson and the Vanderpool’s wore. It made the village feel funny talking to him—just like he was anybody else.