where did Mr. Wilson, Hester pearl and Roger Chillingworth been that night that they would be walking by the scaffolding at midnight

The Reverend Mr. Wilson approaches the scaffold holding a lantern, but only because he is returning from a late-night vigil. He fails to see Dimmesdale, who is standing on the scaffold. Dimmesdale waits a while longer and then bursts out laughing. Much to his surprise, the voice of Pearl answers him.

Hester and Pearl are at the scaffold because they have been at Governor Winthrop's deathbed taking measurements for a robe. Dimmesdale invites them to join him on the stand, which they do. All three hold hands and Pearl asks him, "Wilt thou stand here with Mother and me, tomorrow noontide?" Dimmesdale answers, "I shall, indeed, stand with thy mother and thee one day, but not tomorrow." Pearl persists in her question, and Dimmesdale answers that, "the daylight of this world shall not see our meeting."

At that moment a meteor streaks across the sky, illuminating everything, including Dimmesdale with his hand over his heart and the scarlet letter on Hester's dress. Looking upward, Dimmesdale believes that he sees a giant A in the sky. When he looks down again, Pearl is pointing to Roger Chillingworth, who is watching him from across the street. Chillingworth takes Dimmesdale home.