What is there about Pearl that has caused Dimmesdale "many an alarm"?

Basically Dimmesdale says that he was afraid that Pearl has his features and that the whole town would discover that he had fathered her,

"my own features were partly repeated in her face, and so strikingly that the world might see them! But she is mostly thine!”

Dimmesdale was able to hide his relationship with Hester for much of the novel. By the end he does own up and there are some softer moments.