how does the mariners crew single him out for this deed?

They all knew the Mariner shot the albatross so when things got really bad, they knew where to look.

The crescent moon rose above the ship with "one bright star" just inside its bottom rim, and all at once, the sailors turned towards the Ancient Mariner and cursed him with their eyes. Then all two hundred of them dropped dead without a sound. The Ancient Mariner watched each sailor's soul zoom out of his body like the arrow he shot at the Albatross: "And every soul, it passed me by, / Like the whiz of my cross-bow!"