This is where empathy becomes a good skill to have. I'm certain you can understand the basic setup of the joke even though the exact situation may not be likely in your culture.

The joke isn't even about the time, it's about their anxious thinking.