what does Dr. kimble mean by "...I've seen the time when i might have quarreled with him for it myself.It's too late now, though"?

This surrounds who should get custody of the child. The Doctor says he might have quarreled for it once but he is much too old to raise a child now.