Why did Joey turn down Mrs. Fisher's invitation to stsy at the Fisher home while his own was being fumigated?

Joey won't stay at the Fisher home because of Erik and Arthur. He hasn't even visited since the two intimidated him and insulted his brother.