What is Charlie guilty of in the diner that hurts him most of all?

Charlie is hurt most of all by the realization that he thought the episode in which the young boy dropped the dishes and broke them all over the floor was amusing too. He was ashamed.

I felt sick inside as I looked at his dull, vacuous smile-the wide, bright eyes of a child, uncertain but eager to please, and I realized what I had recognized in him. They were laughing at him because he was retarded. And at first I had been amused along with the rest.

Suddenly, I was furious at myself and all those who were smirking at him. I wanted to pick up the dishes and throw them. I wanted to smash their laughing faces. I jumped up and shouted: "Shut upl Leave him alone! He can't understand. He can't help what he is... but for God's sake, have some respectl He's a human being!