Why were the animals especially afraid of Mr. Frederick from Pinchfield farm?

The animals were afraid of Frederick because of rumors that he planned to destroy the windmill.

At the same time there were renewed rumours that Frederick and his men were plotting to attack Animal Farm and to destroy the windmill, the building of which had aroused furious jealousy in him.