Do you think the loss of diversity is worthwhile?

It very largely depends on the outlook of the character you're viewing the book from. In the eyes of most of that society, it's beneficial because it means everyone is 'equal' and therefore does not experience anything better or worse than any other. Nobody suffers, and therefore life is stable, managed and good.

From the eyes of the Giver and Jonas, the loss of diversity means that the world can't experience great things, because those were sacrificed along with the experience of bad things. It's obviously catastrophic to lose what it feels like to see a rainbow, but it's beneficial to never have to experience the pain of breaking a bone. In order to properly compensate, the society they live in got rid of both, which isn't really seen as good through Jonas' eyes, especially when he wants to rescue Gabriel and show him all that the world has to offer now that he knows.

From an authorial standpoint, the latter (Jonas and the Giver's view) is the one that needs to be considered most. The book itself is a commentary on the potential failures of a utopian and essentially communist society.