What is the real reason behind Troy’s decision not to allow Cory to follow his dreams with sports?

Troy was once a great baseball player in the Negro Leagues, but he was too old to join the Major Leagues when they were integrated. His past mistakes and failures greatly influence his outlook on life and his relationship with his sons. Troy does not want to see his son succeed where he failed,

"The white man ain't gonna let you get nowhere with that football noway."