there is a clock in the first act, but they were not invented yet in Hamlet's time. What is this device called?

This is called an anachronism. It is an inconsistency in some chronological arrangement. In this case, there were clocks in Shakespeare’s time but not in Hamlet's. Shakespeare wrote it in because he thought it helped the plot or suspense in some way.