what does old and young refer to..

I think that you are referring to the line, "What evil looks / Had I from old and young! ..." This illustrates how angry the crew was at him for killing the albatross. Both old and young men wanted to kick the Mariner's ass.