Why is the narrator ambivalent about attacking the men who are evicting the old couple in Invisible Man

He argues that they must follow the rule of law yet he sees the unfairness of the situation. He sees the reflection of generations of white oppression in the eviction. At the same time they will be justifying white sentiments about blacks if they attack the men.