In context, this quote alludes to the fact that the presence of these two people in the same place for the same purpose was unthinkable. That they both wore the same sign of shame was beyond imagination.
While Hester stood in that magic circle of ignominy, where the cunning cruelty of her sentence seemed to have fixed her for ever, the admirable preacher was looking down from the sacred pulpit upon an audience, whose very inmost spirits had yielded to his control. The sainted minister in the church! The woman of the scarlet letter in the market-place! What imagination would have been irreverent enough to surmise that the same scorching stigma was on them both!