how does Zaroff rely on instinct

Actually, it's Rainsford that relies and is saved by instinct when he jumps from the cliff and is presumably dead. Had Zaroff relied on instinct, he may have considered that Rainsford hadn't perished in the fall. Instead, Rainsford wins the game.