What is the significance of the Milky Way to the time traveller?

The Time Traveller notes that, unlike the infinity of time and space, he only has one life. Then he notes that the Milky way has not changed that much or at least "seemed" to not have changed that much. The Milky Way is regarded as part of the Time Traveller's "home".