What had the magistrates been considering regarding Hester'scarlet letter? What was Hester's attitude toward it?

THe Magistrates were considering removing Hester's Scarlet Letter. Hester responds that if she were truly worthy of it being removed, it would have fallen off itself or changed into some other message. Hester is throwing the concept back in their faces. If the letter was a Godly act to brand her then no decision by men should be able to un-brand her.