Throughout Custom House, we see hints of this obsession. The narrator's first finds the package a "riddle" that "strangely interested" him. The elaborate embroidery touches his "sensibilities":
"It seemed to me,--the reader may smile, but must not doubt my word,--it seemed to me, then, that I experienced a sensation not altogether physical, yet almost so, as of burning heat; and as if the letter were not of red cloth, but red-hot iron. I shuddered and involuntarily let it fall upon the floor."
Then deciding to write a whole novel on these pieces of cloth seems to complete his obsession with the story.