"We"
The "We" in the poem (also referred to as "nobody driving by in this car") refers to the speaker of the poem and his/her company driving through Beverly. The poem is spoken from the "we" point of view. "We" aren't as wealthy as the people who live in Beverly. The speaker claims, with a heavy helping of verbal irony, that they are lucky even to get a glimpse into the idyllic lives of the neighborhood.
"These people/they"
The affluent residents of Beverly are referred to in the third person plural, while the speaker and their passengers are referred to in the first person plural. The affluent residents are a point of contrast between the speaker and the people who live in the speaker's neighborhood. The pronoun usage in the poem creates a distinct theme of "us versus them." The residents enjoy fine things, the best material possessions money can buy. Their gardens are beautiful, their tea is luxurious, and even the refuse in their lawns is described as "a neat brilliancy."
"That woman"
In the fourth stanza, the poem briefly zooms in on a single resident of Beverly in her home preparing tea and listening to a record. The quiet moment is interrupted by a reminder that other people need to make livings off the sweat of their backs, and "that woman" is considered privileged, but not without troubles of her own. Her troubles, however, are labeled by the speaker "trouble with a gold-flecked beautiful banner."