What secret does Chillingworth ask Hester to keep?

Chillingworth asks Hester to keep their relationship a secret.

"‘One thing, thou that wast my wife, I would enjoin upon thee,’ continued the scholar. ‘Thou hast kept the secret of thy paramour. Keep, likewise, mine! There are none in this land that know me. Breathe not to any human soul that thou didst ever call me husband!"