Describe the man who is standing on the outside of the crowd?

This quote from chapter 3 pretty much sums his appearance up,

"He was small in stature, with a furrowed visage, which as yet could hardly be termed aged. There was a remarkable intelligence in his features, as of a person who had so cultivated his mental part that it could not fail to mould the physical to itself and become manifest by unmistakable tokens. Although, by a seemingly careless arrangement of his heterogeneous garb, he had endeavoured to conceal or abate the peculiarity, it was sufficiently evident to Hester Prynne that one of this man's shoulders rose higher than the other. Again, at the first instant of perceiving that thin visage, and the slight deformity of the figure, she pressed her infant to her bosom with so convulsive a force that the poor babe uttered another cry of pain. But the mother did not seem to hear it."