After the men leave, Jem reveals his fears to Atticus. What is Jem worried about?

We aren't made aware of the conversation between Atticus and Jem when they left the jail. All we are told is as follows:

Atticus and Jem were well ahead of us, and I assumed that Atticus was giving him hell for not going home, but I was wrong. As they passed under a streetlight, Atticus reached out and massaged Jem’s hair, his one gesture of affection.