What are those factors and how do they contribute?

I think there is an otherness or separateness to race. Blacks and whites are distanced from each other both physically and emotionally. At one point Bigger wants to role-play whites with his friends.

"Let’s play ‘white,’" Bigger said, referring to a game of play-acting in which he and his friends imitated the ways and manners of white folks.

The socio-economic disparity is so large between Black and white that there is little common ground. This leaves an almost forbidden fascination for some of the characters.