What is the dramatic significance of George’s response “I didn’t watch her go”?

In this scene, Curley was purposely attempting to start a fight, but George wasn't taking. George's response was considered indicative of his actions. Had he responded any way, Curley would have felt himself justified in starting a fight because George had watching his wife. His answer confirmed that he was not.