why did meursault visit so infrequently?

Meursault has no emotional attachments to the people in his life. Psychologically, he is incapable of caring about anyone else, and as a person he refuses to pretend he does. Society, the people around him, can't understand this. Even hypocritical and unfelt grief would be preferable to "nothing." His lack of feelings and the way that lack is perceived makes him a societal pariah, damages his reputation even further, and makes those judging his trial lean even more toward finding him guilty.