Why doesn't Chillingworth asssert his rights as Hester's husband?

When Chillingworth saw Pearl it was pretty obvious she had cheated on him. Chillingworth wanted no further relationship with Hester.Really they disliked each other to begin with. He was old and gross and she was young and beautiful. By not revealing his relationship with Hester, Chillingworth gets to do what he loves, torture someone. Chillingworth gets to hang out in the village and find Pearl's father. Then Chillingworth gets to slowly destroy the guy which seems to be how he gets his kicks.