why is Hester in trouble at the beginning?

We know that Herster is in trouble at the beginning of the story because she emerges from the jail wearing the scarlet letter A, which marks her as an adulteress. She is then led to the scaffold to bear her punishment...... public shame. Some of the women in the crowd mark that her sentence would be much tougher had they been the ones to judge her actions.