Light
Light in this poem symbolizes sudden realization, sudden enlightenment one could say, acquisition of memory from a previous life which appears through the senses of smell and sound. This sudden light makes the pieces come together and the certainty in something, love for the person the speaker of the poem addresses, becomes stronger.
Swallow's soar
The meaning of swallow has a few different interpretations. It could symbolize love, which could be seen as the most likely explanation as the poem is about love or confessing love to someone.
But just when at that swallow's soar
Your neck turn'd so
Smell, sound
Motifs of sound and smell are used in connection to the aforementioned déjà vu. The feeling of déjà vu is usually connected to the smells and sounds that take a person back to something they, hypothetically, experienced in a previous life. The familiar sounds and smells take the speaker of the poem back to the past as well that make him experience the sudden light:
I know the grass beyond the door,
The sweet keen smell,
The sighing sound, the lights around the shore.