When the men recognize the voice of Ben Gunn, why aren’t they afraid anymore?

Initially, they believe the voice singing the song is that of Flint's ghost. They were relieved to find it was Ben Gunn, as he ws no threat.