Why does the stranger get angry with Mr Henfrey?

The stranger tried to start a conversation with him, but the Invisible Man got irritated with him and told him to finish his task quickly and get out. Henfrey was very annoyed. On his way, Henfrey thought that the stranger was perhaps wanted by the police as he was all wrapped and had bandaged to conceal his identity.