how does being part of an empire affect a person?

This depends on the person. In Orwell's case, he did not think that the British belonged in Burma. He felt self-conscious of his role as British and the "man in charge" with a gun.