I just installed apt cacher ng for catching my apt upgrade packages and saw a huge time improvement even though I have a good internet connection. It act as a proxy and caches the response packages.

Do you run something similar? Or maybe even run a local repo mirror? Warning, they are really big so I don't think it is recommended unless you really need almost everything.