Why does the narrator say he's invisible?

In the prologue, the narrator first describes what he means by invisible. He is not a ghost or a man with transparent skin. He is invisible by virtue of how others react to him. They do not accept his reality and thus live as though they do not see him. He gives a more direct example by explaining how he almost killed a white man whom he bumped into on the street. He continued to attack the white man as long as the man refused to apologize and kept insulting him. The narrator then realized that the man does not see him as an individual and the narrator walked away laughing at the thought that the man was almost killed by a "figment of his imagination".