Why does Victor not go directly home when he returns to Geneva?

Are you referring to chapter 7? Victor arrives home very late. Instead of going home he spends the evening walking in the woods around the outskirts of the town. He is very perplexed over the death of his brother.