The length of time it takes to buffer videos primarily depends on your internet speed, and the bandwidth and speed of the server you are downloading from.
Apparently, there's a trick to skip the buffering:
Originally posted by another user
When you start up a video, it says Connecting... then as soon as the length of the clip shows up and changes from all 0's then hit the play button again.
If you do that, you do not have to wait for the buffering of the video to reach 100%, and then play automatically. You can initiate it as soon as the time shows up, usually as you reach buffering 0, sometimes before that, when it still says connecting.