What is Garvey’s response to Cole? Do you think it is fair?

Garvey says that they're finished because of Cole's attitude. He tells him that there isn't enough room on the island for both Cole and his attitude, so if he isn't going to do what he needs to do they will tear down the shelter and go home.