Where in the book does it describe the port?

Salem is a port city that failed to mature into a major harbor. The streets and buildings are dilapidated, the townspeople are very sober and old, and grass grows between the cobblestones. The Custom House serves the small ship traffic going through the port, but it is usually a quiet place requiring only minimal work.

Such occasions might remind the elderly citizen of that period, before the last war with England, when Salem was a port by itself; not scorned, as she is now, by her own merchants and ship-owners, who permit her wharves to crumble to ruin, while their ventures go to swell, needlessly and imperceptibly, the mighty flood of commerce at New York or Boston.