What are clues telling us that Armand is 1/2 black?

I really didn't see any clues that Armand is half black until the ending when he finds a letter addressed from his mother to his father. In the letter, his mother thanks God for her husband’s love, but she also reveals that she is grateful that her son will never know that his mother "belongs to the race that is cursed by slavery." Some critics argue that Armand might have had an inkling because his treatment of Desiree, after the baby looks mixed, is so abusive and cruel. I don't think this is the case: I think he is just cruel and racist.