By being invisible, the narrorator means that other refuse to see him for who he truly is. They only see what they want to see. In the prologue, Ellison used invisibility to show how the narrator felt: Unseen by those around him. He described how it was as if people were only seeing what was around him (his outward appearance) and not what was within that "bubble" they felt they did not need to pop (the inside). This refusal to pop the "bubble" seperating the narrator and other people around him was his invisibility.
Invisibility is an appropriate work for some African Americans. Many were ignored, and separated from the rest of society. But not all of the African America experience was this way. People like Martin Luther King were hardly invisible. Many white Americans could see him clearly; who he was and what he stood for. So, it was not a matter of invisibility at this point, but merely a matter of refusing to accept the fact that someone was trying to make a difference.
This is much like what happened to the narrator. He was "invisilbe" during his days in college. When Dr. Bledsoe and Mr. Norton could not accept the fact that he was trying to make a change (by showing Mr. Norton "hidden" parts of black society), they felt threatened. The narrator went from being invisible to being a threat. People are still not seing him for who he truly is, but they no longer treated him like a test subject.
So, not all of African American history was based on invisiblity. Much of it consisted of non-acceptance and being thought of as a threat to others, nad thier own selfish demenors.