When Pip returns to Satis House to visit Estella and Miss Havisham, why doesn't he visit Joe?

Pip doesn't want to visit Joe and Biddy because he doesn't want to be associated with the forge. He is basing his decisions on what he believes society deems proper.... and now that he's a gentleman, Pip doesn't want to be linked with his past. Note, this decision does leave him feeling incredibly guilty.... as it should.