Why did Captain Smollett complain to Dr. Livesey and the Squire?

The Captain complained because he was left ignorant of reasons for the voyage and unable to choose his own crew. Even worse, everyone around knew the task at hand.... except for him.

"I was engaged, sir, on what we call sealed orders, to sail this ship for that gentleman where he should bid me," said the captain. "So far so good. But now I find that every man before the mast knows more than I do. I don't call that fair, now, do you?"

"No," said Dr. Livesey, "I don't."

"Next," said the captain, "I learn we are going after treasure—hear it from my own hands, mind you. Now, treasure is ticklish work; I don't like treasure voyages on any account, and I don't like them, above all, when they are secret and when (begging your pardon, Mr. Trelawney) the secret has been told to the parrot."

"Silver's parrot?" asked the squire.

"It's a way of speaking," said the captain. "Blabbed, I mean. It's my belief neither of you gentlemen know what you are about, but I'll tell you my way of it—life or death, and a close run."