How did Silver and Jim know the treasure had already been taken?

I don't think Silver knew the treasure had already been taken. According to the text, he was surprised and disappointed;

There never was such an overturn in this world. Each of these six men was as though he had been struck. But with Silver the blow passed almost instantly. Every thought of his soul had been set full-stretch, like a racer, on that money; well, he was brought up, in a single second, dead; and he kept his head, found his temper, and changed his plan before the others had had time to realize the disappointment.

I will need to look further for information pertaining to Jim.