In Chapter 4, why doesn’t Hiram’s dad Harlan want Hiram in Greenwood this summer?

Grandpa gives Harlan a difficult time about moving.... not merely difficult, but continuous as well. Grandpa thinks that Harlan is taking his grandson away.... and from the text, we can infer that Harlan wants to keep Hiram away from his grandfather's influence and the racism in the South.