What does the owl-eyed man in the library find extraordinary about Gatsby's library?

The books are real books, not just the front cover for show.

In 1920's when a book was binded the pages were still stuck together, so when somebody has read the book they would have to tear the pages apart hence leaving jagged edges.

The owl-eyed man notices that the books have not been read at all, this is the irony in the books, they are as good for show only.