What experiment does the man perform? Why is he suprised at the result?

The sight of the dog put a wild idea into his head. He remembered the tale of the man, caught in a blizzard, who killed a steer and crawled inside the carcass, and so was saved. He would kill the dog and bury his hands in the warm body until the numbness went out of them. Then he could build another fire.

The entire episode with the dog only encompasses a few paragraphs; they're wonderfully written. So if you're required to give a more detailed answer, I'd go with the text.

"The sight of the dog put a wild idea into his head. He remembered the tale of the man, caught in a blizzard, who killed a steer and crawled inside the carcass, and so was saved. He would kill the dog and bury his hands in the warm body until the numbness went out of them. Then he could build another fire. He spoke to the dog, calling it to him; but in his voice was a strange note of fear that frightened the animal, who had never known the man to speak in such way before. Something was the matter, and its suspicious nature sensed danger—it knew not what danger, but somewhere, somehow, in its brain arose an apprehension of the man. It flattened its ears down at the sound of the man's voice, and its restless, hunching movements and the liftings and shiftings of its forefeet became more pronounced; but it would not come to the man. He got on his hands and knees and crawled toward the dog. This unusual posture again excited suspicion, and the animal sidled mincingly away.

The man sat up in the snow for a moment and struggled for calmness. Then he pulled on his mittens, by means of his teeth, and got upon his feet. He glanced down at first in order to assure himself that he was really standing up, for the absence of sensation in his feet left him unrelated to the earth. His erect position in itself started to drive the webs of suspicion from the dog's mind; and when he spoke peremptorily, with the sound of whip-lashes in his voice, the dog rendered its customary allegiance and came to him. As it came within reaching distance, the man lost his control. His arms flashed out to the dog, and he experienced genuine surprise when he discovered that his hands could not clutch, that there was neither bend nor feeling in the fingers. He had forgotten for the moment that they were frozen and that they were freezing more and more. All this happened quickly, and before the animal could get away, he encircled its body with his arms. He sat down in the snow, and in this fashion held the dog, while it snarled and whined and struggled.

But it was all he could do, hold its body encircled in his arms and sit there. He realized that he could not kill the dog. There was no way to do it."