What are some reason the townspeople don't think Emily's relationship with Homer is a good idea?

From the text:

"When she had first begun to be seen with Homer Barron, we had said, "She will marry him." Then we said, "She will persuade him yet," because Homer himself had remarked--he liked men, and it was known that he drank with the younger men in the Elks' Club-- that he was not a marrying man. Later we said, "Poor Emily" behind the jalousies as they passed on Sunday afternoon in the glittering buggy, Miss Emily with her head high and Homer Barron with his hat cocked and a cigar in his teeth, reins and whip in a yellow glove.

Then some of the ladies began to say that it was a disgrace to the town and a bad example to the young people. The men did not want to interfere, but at last the ladies forced the Baptist minister--Miss Emily's people were Episcopal-- to call upon her. He would never divulge what happened during that interview, but he refused to go back again. The next Sunday they again drove about the streets, and the following day the minister's wife wrote to Miss Emily's relations in Alabama."