We know he's divorced, we know he's been separated from his daughter, we know he's terrified that something will happen to separate them again. He's afraid of his ex-wife, not for what she can do yo him physically, but for the emotional turmoil he's endured.
We can see he loves his daughter, although their separation has left him uncomfortable with their time together. Everything they do from play in the dirt to walking through the park holds an opportunity for something to go wrong. He's always on his guard, and it takes away from what should be a happy time.