Why does Dimmesdale think his and Hester's sin isn't the worst in the world? How does Hester describe her sin?

Hester believes her own worse sin to be her hatred for Chillingworth. She considered her worse crime to have allowed him to hold her hand.

"She marvelled how she could ever have been wrought upon to marry him! She deemed it her crime most to be repented of, that she had ever endured, and reciprocated, the lukewarm grasp of his hand, and had suffered the smile of her lips and eyes to mingle and melt into his own."

Chillingworth commits crimes worse than adultery. he wants to harm Dimmesdale and eventually Hester, where as Hester and Dimmesdale just feel in love.